Nerderati

Simplify Your Life With an SSH Config File

2012-10-20T18:57:51Z

If you’re anything like me, you probably log in and out of a half dozen remote servers (or these days, local virtual machines) on a daily basis. And if you’re even more like me, you have trouble remembering all of the various usernames, remote addresses and command line options for things like specifying a non-standard connection port or forwarding local ports to the remote machine.

Shell Aliases

Let’s say that you have a remote server named dev.example.com, which has not been set up with public/private keys for password-less logins. The username to the remote account is fooey, and to reduce the number of scripted login attempts, you’ve decided to change the default SSH port to 2200 from the normal default of 22. This means that a typical command would look like:

$ ssh fooey@dev.example.com -p 22000
password: *************

Not too bad.

We can make things simpler and more secure by using a public/private key pair; I highly recommend using ssh-copy-id for moving your public keys around. It will save you quite a few folder/file permission headaches.

$ ssh fooey@dev.example.com -p 22000
# Assuming your keys are properly setup...

Now this doesn’t seem all that bad. To cut down on the verbosity you could create a simple alias in your shell as well:

$ alias dev='ssh fooey@dev.example.com -p 22000'
$ # To connect:
$ dev

This works surprisingly well: Every new server you need to connect to, just add an alias to your .bashrc (or .zshrc if you hang with the cool kids), and voilà.

~/.ssh/config

However, there’s a much more elegant and flexible solution to this problem. Enter the SSH config file:

# contents of $HOME/.ssh/config
Host dev
HostName dev.example.com
Port 22000
User fooey

This means that I can simply $ ssh dev, and the options will be read from the configuration file. Easy peasy. Let’s see what else we can do with just a few simple configuration directives.

Personally, I use quite a few public/private keypairs for the various servers and services that I use, to ensure that in the event of having one of my keys compromised the dammage is as restricted as possible. For example, I have a key that I use uniquely for my github account. Let’s set it up so that that particular private key is used for all my github-related operations:

Host dev
HostName dev.example.com
Port 22000
User fooey

Host github.com
IdentityFile ~/.ssh/github.key

The use of IdentityFile allows me to specify exactly which private key I wish to use for authentification with the given host. You can, of course, simply specify this as a command line option for “normal” connections:

$ ssh -i ~/.ssh/blah.key username@host.com

but the use of a config file with IdentityFile is pretty much your only option if you want to specify which identity to use for any git commands. This also opens up the very interesting concept of further segmenting your github keys on something like a per-project or per-organization basis:

Host github-project1
User git
HostName github.com
IdentityFile ~/.ssh/github.project1.key

Host github-org
User git
HostName github.com
IdentityFile ~/.ssh/github.org.key

Host github.com
User git
IdentityFile ~/.ssh/github.key

Which means that if I want to clone a repository using my organization credentials, I would use the following:

$ git clone git@github-org:orgname/some_repository.git

Going further

As any security-conscious developer would do, I set up firewalls on all of my servers and make them as restrictive as possible; in many cases, this means that the only ports that I leave open are 80/443 (for webservers), and port 22 for SSH (or whatever I might have remapped it to for obfuscation purposes). On the surface, this seems to prevent me from using things like a desktop MySQL GUI client, which expect port 3306 to be open and accessible on the remote server in question. The informed reader will note, however, that a simple local port forward can save you:

$ ssh -f -N -L 9906:127.0.0.1:3306 coolio@database.example.com
# -f puts ssh in background
# -N makes it not execute a remote command

This will forward all local port 9906 traffic to port 3306 on the remote dev.example.com server, letting me point my desktop GUI to localhost (127.0.0.1:9906) and have it behave exactly as if I had exposed port 3306 on the remote server and connected directly to it.

Now I don’t know about you, but remembering that sequence of flags and options for SSH can be a complete pain. Luckily, our config file can help alleviate that:

Host tunnel
HostName database.example.com
IdentityFile ~/.ssh/coolio.example.key
LocalForward 9906 127.0.0.1:3306
User coolio

Which means I can simply do:

$ ssh -f -N tunnel

And my local port forwarding will be enabled using all of the configuration directives I set up for the tunnel host. Slick.

Homework

There are quite a few configuration options that you can specify in ~/.ssh/config, and I highly suggest consulting the online documentation or the ssh_config man page. Some interesting/useful things that you can do include: change the default number of connection attempts, specify local environment variables to be passed to the remote server upon connection, and even the use of * and ? wildcards for matching hosts.

I hope that some of this is useful to a few of you. Leave a note in the comments if you have any cool tricks for the SSH config file; I’m always on the lookout for fun hacks.

Ideas of March

2011-03-18T02:11:48Z

Four years ago when I started using a micro-blogging service, I revelled in the sheer simplicity and low barrier to communicating my thoughts. Blogging, in that era, seemed like a historical vestige on the verge of being consumed and overtaken by the rapid fire, real-time tweets of my peers. I jumped on that bandwagon, and never looked back. Until now.

Like my friend Chris (and many others), I am reminded of the time when we used to have productive and lengthy discussions in and around blog posts. Moreover, I am reminded of the fact that when I truly need to find some snippet of information or tutorial on a particular web-related topic, the solution is rarely found in multiples of one hundred forty characters.

Now, I’m not ditching Twitter. Micro-blogging does serve a very important role (and I was reminded of this during the recent tsunami/earthquake in Japan) in the effective transmission of near real-time information, but I miss the asynchronous nature of blogs and the less ephemeral nature of the community that surrounds them. It’s about time that I do something to remedy that.

For the rest of the month of March, I will be posting at least two articles per week on various different tech-related topics, with the intent on starting a habit of regular, informative blogging, and I hope that you will join me in this endeavour. I hope and believe that the long term rewards will far outweigh the short term investment in time and effort.

Let’s try and move the conversations we have from being limited by the bits and bytes of tweets to sentences and paragraphs.

Code Your Art Out: If Non-Profits and Tech Had Babies

2011-03-11T21:23:12Z

Ever since I had the pleasure of keynoting at Make Web Not War: For The Web in 2010, I’ve been eagerly awaiting the announcement of the 2011 edition, which was announced just a few days ago.

The title of this year’s competition is Code Your Art Out, and the theme is that of “helping non-profits harness the power of technology in order to better serve its communities and members”. Oddly enough, this is a topic that I have discussed many times over the years with @katiestardust, and is thus very close to my heart.

The competition itself is rather simple: You register, think of an application that might be useful for non-profits (there’s a list of ideas available if you’re stuck), and the architect/code/design/innovate to your heart’s content. As an added incentive, there’s a non-negligible chunk of cash given out to the teams that place in first and second. The only technological requirements are that you need to use either the Windows Azure clould platform, or deploy on Windows IIS as your webserver.

While there’s no requirement that the applications developed be made available under an open source license, I’m hopeful that many submissions will choose to go this route.

The use of modern web technology stacks and applications in most non-profits is woefully lacking, and the space is ripe for innovation. I’m very excited to see what applications people develop!

A few resources:

Twitter account
Facebook Event page
Facebook Page

Perpetual IRC – The Proxy Edition

2011-04-05T14:17:48Z

Last time, we looked at how one could maintain a persistent session in IRC through the use of a terminal multiplexer (such as screen or tmux) and SSH. While this has the advantage of being very easy to setup, there are a few very obvious disadvantages and trade-offs:

You must be on a machine that is able to SSH into your remote server.
You are confined to using the command-line IRC client running in the remote session.
Any scripts that attempt to interact with your ‘local’ desktop (e.g. Growl notifications), are painful to setup, if not impossible.

For many, including myself, the first two are deal-breakers. I use a variety of computing devices on a daily basis, and asking me to SSH into a remote server from my smartphone to then attach a screen session is not reasonable. Luckily, there does exist a method for establishing a ‘perpetual’ IRC session that does not suffer from any of the disadvantages brought on by the SSH + multiplexer approach.

Enter Bip

Bip is an IRC proxy by Arnaud Cornet and Loïc Gomez, and is released under the GPLv2 license. From their about page:

[Bip keeps you] connected to your preferred IRC servers, can store the logs for you, and even send them back to your IRC client(s) upon connection. You may want to use bip to keep your logfiles (in a unique format and on a unique computer) whatever your client is, when you connect from multiple workstations, or when you simply want to have a playback of what was said while you were away.

Strangely, Bip does not seem to be very well known amongst people I would call “heavy” IRC users, despite it’s very lengthy list of useful features and the relatively simple and straightforward setup.

Remote Server Setup

It may be possible to install Bip through a package manager, but we’ll go right to building from source to ensure that everyone is on the same page. Feel free to skip this part if you are able to install bip through your distribution’s package manager.

Note: You will need to have git, make, gcc, lex and yacc (the latter two are for lexing/parsing the bip configuration file) to compile from source.

$ mkdir src && cd src
$ git clone http://rcs-git.duckcorp.org/projects/bip/bip.git
$ cd bip
$ ./bootstrap
$ ./configure # Use the --prefix option to install somewhere other than /usr/local
$ make && make install

Assuming we have the proper dependencies installed and there were no errors, we should have two binaries — bip and bipmkpw — installed in /usr/local/bin/ (or whatever path was specified for the ‐‐prefix option during the configure phase. In case you decided to skip the make install part of the above instructions, the compiled binaries are available in the src folder of your bip source directory. Place these binaries in an appropriate location, preferably reachable via $PATH.

Configuration

Note: Nearly all of this configuration information can be found in the README file included with bip.

For this configuration file to work, we need to create a directory to store a few things:

$ mkdir -p ~/.bip

Let’s start off with the most bare-bones configuration file possible. Create a file named bip.conf in your $HOME/.bip/ directory with the following contents:



# Remember to change out $HOME for your home directory (e.g. `/home/joel`) below.
ip = “0.0.0.0″;

port = 6667;

log_level = 2;

log_root = “$HOME/.bip/logs”;

log_sync_interval = 5;
# Network definition, a name and server info

network {

	name = “freenode”;

	server { host = “irc.freenode.net”; port = 6667; };

};
user {

	name = “joel”;

	password = “
“;

	default_nick = “jperras”;

	default_user = “jperras”;

	default_realname = “Joël”;
	backlog = true;		        # enable backlog

	backlog_lines = 200;		# number of lines in backlog, 0 means

					# no limit

	backlog_always = false;		# backlog even lines already backlogged

	backlog_reset_on_talk = false;

	backlog_reset_connection = false;

	backlog_msg_only = true;
	connection {

		name = “primary”;		# used by bip only

		network = “freenode”;	# which ircnet to connect to
		# Autojoined channels:

		channel { name = “#awesome-channel”; };

		channel { name = “#secret-channel”; key = “passw0rd”; };

		channel { name = “#unimportant”; backlog = false; };

	};

};

Alright, let’s go through what these options actually do.

ip = "0.0.0.0";
port = 6667;

The ip string indicates from which IP bip should accept incoming connections, and port is self-explanatory. By setting it to 0.0.0.0, bip will accept a connection from any IP, which is most likely what you want. Additionally, restricting incoming connections to certain IP addresses for particular ports is something that can (and should) be pushed to your firewall.

Note: Be sure that you have left an opening in your firewall for connecting to your proxy!

log_level = 2;
log_root = "$HOME/.bip/logs";
log_sync_interval = 5;

A log_level of 2 indicates that we want to log all errors and warnings, but ignore info and debug messages. The log_root indicates where logs should be stored, and log_sync_interval dictates at what time interval the log file is written to disk.

network {
name = "freenode";
server { host = "irc.freenode.net"; port = 6667; };
};

Next, we define a network that we would like to connect to. You can specify an arbitrary name — I could have just as easily called this network opensource — and the relevant connection information in the server key.

user {
name = "joel";
password = "<some_long_password_hash>";
default_nick = "jperras";
default_user = "jperras";
default_realname = "Joël";
# ...

Here, we define a bip user named “joel” (this name is only used when you connect from your local client to your proxy), and a password. The password that you insert here should be created through the bipmkpw binary, which will salt & hash an input string to produce the value that needs to be placed in the above user {} block. The other three options of default_nick, default_user, and default_realname should be self-evident if you have ever used IRC before.

backlog = true;
backlog_lines = 200;
backlog_always = false;
backlog_reset_on_talk = false;
backlog_reset_connection = false;
backlog_msg_only = true;

Now we start getting into the more interesting configuration options for bip. backlog = true enables backlogging, which means that when your local client (e.g. smartphone, desktop client, web client) reconnects to your proxy, all the messages that have been logged since your last connection will be replayed to you. This is, in my opinion, the best reason to consider using an IRC proxy such as bip; you’ll never miss a message, conversation or context again.

backlog_lines = 200 sets an upper limit to the number of messages per channel that will be sent to your client upon reconnection.

backlog_always = false indicates that you don’t want to receive backlog messages that you have already seen. By setting this option to true, every time you connect a client to your proxy you will receive backlog_lines of backlog messages for each channel. If you do enable this option (which comes in handy when you regularly connect via two or more devices), make sure the value for backlog_lines is something reasonable.

backlog_reset_on_talk and backlog_reset_connection indicate that the backlog marker should be reset every time you talk, scoped to the particular channel or entire connection, respectively.

By setting backlog_msg_only = true, we only log messages and ignore all IRC events, including joins/parts/quits/nick changes, and the like.

On to the connection definition:

connection {
name = "freenode"; # used by bip only
network = "freenode"; # which IRC network to connect to

# Autojoined channels:
channel { name = "#awesome-channel"; };
channel { name = "#secret-channel"; key = "passw0rd"; };
channel { name = "#unimportant"; backlog = false; };
};

First, we give a name to the connection (which will be useful in the following segment), and specify which network (defined previously) that we wish to connect to.
Finally, we specify the channels that we wish to automatically connect to when first starting up the bip proxy. Note the use of key for password protected channels, and the backlog = false usage for channels that we never want to receive a backlog for.

Note that these channels are auto-joined when the bip daemon starts, and not each time your local client connects to the proxy. Once bip has started up and we connect to it, we are free to join/part channels at will, and these will be persisted between client connections.

Starting The Server

Alright, let’s start the bip process:

$ /usr/local/bin/bip -f /path/to/bip.conf

Note: If things don’t seem to be working as expected, tail’ing the bip log file may yield some clues as to what is going wrong.

The bip process will daemonize itself by default; use the -n option to have it remain in the foreground, if so desired.

When you start the bip process, it initiates the IRC handshake & dance that normally occurs when you connect a local client to an IRC network. The channels that you defined in the bip configuration file are now connected to and being kept open by bip.

Connecting to the Proxy

Now that we’ve started the bip proxy on our remote server, let’s fire up an IRC client and configure the server connection information.

Examples

I’ve included a few examples of how the server connection information should look like (given the example bip configuration that I’ve provided above), in both a GUI-based client as well as a text-based one.

Important: The password that must be used when connecting to your proxy must follow this pattern:

username:unHashedPassword:network

Where the username portion corresponds to the value of the name field in the user { } block in your bip configuration, the unHashedPassword portion is the un-hashed version of the password that you generated (using bipmkpw) and pasted in your config, and network is the name of the IRC network (also defined in your configuration) that you wish to connect to.

An interesting thing to note is that, with this particular scheme, it’s quite simple to connect to a different network, or to add a new user to the proxy.

Colloquy

Irssi

servers = (
{
address = "my-proxy-host.com";
chatnet = "Proxy";
port = "7778";
password = "joel:myHashedPassword:freenode";
autoconnect = "yes";
}

Conclusion

And that’s it! We are now perpetually connected to IRC, and can connect to our proxy using an unlimited number of devices in a completely transparent and seamless manner. Moreover, the logs for all channels that we are connected to are saved and automatically rotated on our remote server, making for an easy to maintain archive in the absence of another external channel logging mechanism. Additionally, we can now generate an OpenSSL certificate for our remote server, and configure bip as well as our local clients to communicate with our proxy over a secured connection. This, along with many other features and options that I did not cover, are all documented in the README and man pages that are installed alongside the bip binary; Please take a look for more information on using bip to it’s full potential.

Perpetual IRC – The Multiplexer Edition

2010-09-07T15:04:35Z

As is the case with most people involved in Open Source, I’m on IRC all day long. I can help people from around the world use some of the projects I’ve helped create, as well as some of the software that I use on a daily basis.

Additionally, IRC is also the preferred form of group communication for the company I work for. One of the major advantages that IRC has over your ‘traditional’ instant messenger clients is that, with a minimum amount of effort and hardware, you can create a setup that will remain perpetually^* connected, even when you’re not online. That means that you can keep logs of conversations, receive messages, catch up on what the current topic of discussion is when starting your day, and still be able to shut down your computer at night if you so choose.

There are a few ways of achieving the status of perpetual IRC denizen. Note, however, that almost any method requires that you have access to a remote server in addition to your local machine. It doesn’t need to be anything special – a cheap VPS will do just fine, as long as you can install software packages and open the necessary ports in the firewall.

Simplest. If you want to expend the least amount of effort, at the cost of some flexibility, the easiest method of achieving IRC immortality is with the use of a terminal multiplexer, such as screen or tmux (my personal favourite), and a console-based IRC client, such as Irssi.

Don’t know what a terminal multiplexer is? Here’s a description, right from the screen homepage:

Screen is a terminal multiplexer: it takes many different running processes and manages which of them gets displayed to the user. Think of it as a window manager for your console or terminal emulator. With screen, you can have interactive processes running on your home computer and connect to them from anywhere else to check on their progress.

Let’s assume that you’ve installed screen and Irssi on your remote server. What we want to do is start Irssi inside a screen session:

$ screen -S irc irssi # name the session `irc` and start `irssi` in it

Digression. At this point, feel free to browse the documentation on how to properly setup your preferred servers, channels and other configuration parameters for the IRC client. For a quick test, you can try the following at the Irssi prompt:

/set nick YourNickName
/connect irc.freenode.net
/join #some-interesting-channel

You can also automate the process, so that upon startup Irssi will auto-connect to the servers and channels you specify:

/server add -auto -network Freenode irc.freenode.net 6667
/network add -nick YourNickName Freenode
/channel add -auto #some-interesting-channel Freenode

If you have a registered nickname, you can have it auto-identify as well:

/network add -autosendcmd "/msg nickserv identify your_pasword_here ;wait 2000" Freenode

And you’ll probably want to cut down on the noise:

/ignore #some-interesting-channel ALL -PUBLIC -ACTIONS

At this point, you should have a fully functioning IRC client inside of a screen session. You can detach from the current session by pressing control-a d, and re-attach with:

$ screen -r

See where this is going? With this setup, you can simply maintain your IRC connection within a screen session on your remote server. When you want to “log on”, you simply SSH into your remote server, screen -r, and voilà! Perpetual IRC.

This approach, while simple, has some very obvious drawbacks:

You must be on a machine that is able to SSH into your remote server.
You are confined to using the command-line IRC client running in the remote session.
Any scripts that attempt to interact with your ‘local’ desktop (e.g. Growl notifications), are painful to setup, if not impossible.

Okay, so it’s not completely ideal. But it does the job – as long as your remote screen session remains operational, you’ll be logged in to IRC.

In my next post, we’ll look at how you can achieve the same results, but without any of the aforementioned drawbacks, through the use of a specialized IRC proxy daemon.

^*: Of course, if the remote server running the terminal multiplexer session goes down, it’s not really ‘perpetual’.

Making Git Behave

2010-08-04T17:08:03Z

It’s no secret that I’m a big fan of Git, and of distributed version control in general; they offer a compelling toolset and degree of flexibility that you would be hard pressed to find in a “traditional” centralized version control system. Instead of discussing the merits of a DVCS or performing feature comparisons between particular implementations — there are enough of those already — let’s take a look at how you can bend Git to your will through configuration, and a few useful aliases. If you’ve not encountered this before, you’ll be surprised at how much can be accomplished with just a few lines in a configuration file.

Your .gitconfig

This is where the magic happens. Git, by default, will attempt to find a .gitconfig file in three places:

$GIT_DIR/config for per-repository configurations
~/.gitconfig for user-specific configurations
$(prefix)/etc/gitconfig for system-wide configurations

Read up on the various options and overrides that git-config accepts, as well as alternative methods to interactively add/remove/replace configuration options.

For the rest of this article we’ll be adding configurations to ~/.gitconfig, but there’s nothing stopping you from doing it in a per-repository (or even system-wide) configuration file instead.

For reference, here is a simplified version of the config I use:

[color]
diff = auto
status = auto
branch = auto
[user]
name = Joël Perras
email = joel.perras@gmail.com
[status]
relativePaths = false
[core]
excludesfile = /Users/joel/.gitignore
[alias]
d = diff
dc = diff --cached
lc = log ORIG_HEAD.. --stat --no-merges
gl = log --oneline --decorate --stat --graph
st = status -sb
prune-all = !git remote | xargs -n 1 git remote prune
whatis = show -s --pretty='tformat:%h (%s, %ad)' --date=short
whois = "!sh -c 'git log -i -1 --pretty=\"format:%an <%ae>\n\" --author=\"$1\"' -"

Also available as a Gist.

Now let’s go through what these configuration options actually do.

[color]

[color]
diff = auto
status = auto
branch = auto

These were, in all likelihood, the first configuration options I setup once I had installed Git for the first time. I work in a terminal window all day, and having proper and consistent colourization makes all the difference when attempting to quickly determine the state of your repository:

With these options set, all diff, status and branch commands will have (where appropriate) colourized output.

[user]

If you ever plan on making your repository available to other developers, these options are essential for identifying who you are. They’re pretty self-explanatory:

[user]
name = Joël Perras
email = joel.perras@gmail.com

[status]

The sole configuration I have under this segment is something of a personal preference. When I’m four folders deep into a repository and I invoke git status, I don’t need to see where the changed/added/deleted files are relative to where I invoked the command — I’m far more interested in their absolute location relative to the repository root. That’s where relativePaths = false comes in handy:

[status]
relativePaths = false

[core]

There are some files you just never want to commit, no matter what project you’re working on. As an avid Vim user, I sometimes have a few *.swp files lying around (mostly due to my own fault), which have no place in any project history. Instead of adding an ignore rule to each and every repository, I added this configuration directive to my ~/.gitconfig, indicating where my global never-track-any-of-these-file-patterns exists:

[core]
excludesfile = /Users/joel/.gitignore

[alias]

Now we get to the interesting stuff. Aliases in Git are exactly what you expect them to be: user-defined shortcuts that group together pre-existing git (and sometimes shell) commands and their options.

Edit: As some people have pointed out, there are a few flags/commands that will not work unless you have git 1.7.2 or above installed. Pardon the confusion.

[alias]
d = diff
dc = diff --cached
lc = log ORIG_HEAD.. --stat --no-merges
gl = log --oneline --decorate --stat --graph
st = status -sb
serve = !git daemon --reuseaddr --verbose --base-path=. --export-all ./.git
prune-all = !git remote | xargs -n 1 git remote prune
whatis = show -s --pretty='tformat:%h (%s, %ad)' --date=short
whois = "!sh -c 'git log -i -1 --pretty=\"format:%an <%ae>\n\" --author=\"$1\"' -"

Some explanations:

git d and git dc

I’m often taking diffs between different revisions in projects, so these two are quite heavily used.

git lc

Lists all new commits after you fetched. This is especially useful when working on a very active project with multiple developers.

git gl

This is the default command that I invoke when I wish to inspect the log. The output is concise as well as informative:

git st

My preferred method of viewing the status of a repository, which gets rid of the fluff that usually accompanies the help-type text of git status:

git prune-all

Since git remote prune doesn’t know how to prune all of your remotes at once, let’s teach it that trick. This is especially useful if you’re tracking many remotes from various contributors to a project where the branches get nuked/renamed on a regular basis.

git whatis <commit-ish>

Pass this command a hash, branch name or tag, and it will give you a one-liner containing the commit message associated to it as well as the date it was committed on. Useful when git show is more information than you need.

git whois <name|email>

Another one of those useful commands when you’re working on a project with a multitude of collaborators, and need to fish out an email address when all you remember is part of their name, or vice versa. Note that if the collaborator in question never configured their name and/or email address (as described above), then this command will not be very useful.

And that’s it! With nothing more than a few lines in a configuration file, we’ve managed to make Git be more informative as well as more concise.

If you have any tips/tricks/configs that you can’t live without, leave a comment below.

Quite a few of the aliases and what I would consider “sane” configuration options have been lifted from the official wiki for Git. I highly recommend perusing it, once you’ve become familiar with Git itself.

Compiling Vim With Ruby Integration On Snow Leopard

2010-08-24T13:51:17Z

One of my favourite plugins for Vim is Command-T:

an extremely fast, intuitive mechanism for opening files with a minimal number of keystrokes. It’s named “Command-T” because it is inspired by the “Go to File” window bound to Command-T in TextMate.

Sadly, the default installation of Vim on Snow Leopard does not have support for the ruby interpreter compiled in, which is a pre-requisite for using the plugin. Luckily, that’s easy enough to remedy, and in the process we’ll learn a thing or two about compiling your own custom Vim binary.

Let’s start off by getting the source code from the official Mercurial repository:

$ hg clone https://vim.googlecode.com/hg/ vim
$ cd vim

(Note: You don’t need to use the Mercurial repository – there are mirror sources for Subversion, CVS, as well as good ol’ tarballs with patches.)

The default branch for the mercurial repository contains the code for Vim 7.2 at the time of this writing. There is also a vim73 branch available for those feeling a bit more adventurous and wishing to compile the beta release of the next version. For this article, we’ll be sticking to the stable 7.2 release in the default branch.

Now, let’s take a look at the possible configuration options:

$ ./configure --help

There are quite a few, and I suggest that you take the time to read through them – most are quite self-explanatory. For Command-T, the one that we are interested in is the

--enable-rubyinterp

option.

So let’s take a shot at the simplest installation for terminal-based Vim usage, one without the GUI interface and (Linux) mouse daemon support:

$ ./configure --prefix=/my/install/prefix --enable-rubyinterp --enable-gui=no --disable-gpm
$ make

After the compilation process finishes (presumably with no errors), the first thing you’ll want to do is ensure that the binary you just built functions as expected:

$ ./src/vim --version | grep ruby
# you should see a `+ruby` line entry
$ ./src/vim

If you see the +ruby entry in the –version output and the binary launches without any errors, rejoice in your own awesomeness. That’s all there is to it.

If, however, you see something similar to this:

Vim: Caught deadly signal SEGV
Vim: Finished.
zsh: segmentation fault ./src/vim

you’ve probably fallen prey to a (currently) not very well documented issue: Vim 7.2 does not support the integration of Ruby 1.9.x on Snow Leopard.

This means that if you’ve used a package manager such as Homebrew, MacPorts or Fink (shudder) to install the latest version of Ruby, Vim will link to that latest version instead of the system default installation of ruby 1.8.7.

Let’s fix that.

We’re going to edit the src/auto/config.mk generated by configure that was run earlier. Note that if you re-run configure at a later time, your changes to config.mk will be lost.

Find the lines that look like this:

RUBY = /usr/local/bin/ruby
RUBY_SRC = if_ruby.c
RUBY_OBJ = objects/if_ruby.o
RUBY_PRO = if_ruby.pro
RUBY_CFLAGS = -I/usr/local/Cellar/ruby/1.9.1-p378/include/ruby-1.9.1 -I/usr/local/Cellar/ruby/1.9.1-p378/include/ruby-1.9.1/i386-darwin10.4.0 -DRUBY_VERSION=19
RUBY_LIBS = -lruby -lpthread -ldl -lobjc

(Note: Your specific paths and/or versions may differ depending on the package manager that you are using. The above paths are actually not important, however, since we actually want to reset them to the system defaults.)

and replace them with the following:

RUBY = /usr/bin/ruby
RUBY_SRC = if_ruby.c
RUBY_OBJ = objects/if_ruby.o
RUBY_PRO = if_ruby.pro
RUBY_CFLAGS = -I/System/Library/Frameworks/Ruby.framework/Versions/1.8 -I/usr/lib/ruby/1.8/universal-darwin10.0
RUBY_LIBS = -framework Ruby

Alright, let’s see if this worked.

$ make clean && make

Before we check the binary as we did before, let’s see if we linked to the correct ruby libraries:

$ otool -L src/vim
src/vim:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.0)
/usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/libruby.1.dylib (compatibility version 1.8.0, current version 1.8.7)
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 44.0.0)
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 550.29.0)

Looking good so far – the binary is linked to the framework version of Ruby that comes as a default on Snow Leopard.

Let’s do a version check:

$ ./src/vim --version | grep ruby
# you should see a `+ruby` line entry
$ ./src/vim

And voilà: A custom-built Vim with ruby integration that will happily run the Command-T plugin.

All that’s left is to install it:

$ make install

Assuming your PATH is setup to correctly find the new Vim binary, you should be all set.

WebNotWar

2010-05-20T21:41:30Z

I’m quite happy to announce that I will be giving the keynote address at this year’s WebNotWar/For The Web conference, taking place on May 27th, 2010.

First, it’s a free conference for attendees. Being an open-source kinda guy, I know & love the word free. Second, the theme of the conference is ‘interoperability’, which is something very important to me, to my open source work, to my job, and to end users (I like to call them ‘normies’) – so much so that my keynote will address this directly.

The technological landscape of the web is diverse, and keeping track of new developments and projects is a full time job. Luckily, conferences like this one provide a great way to meet the people that make the web of tomorrow, and get a head start on what the next Big Thing is going to be.

I hope to see you there!

The Wonders and Simplicity of Redis Sets

2010-02-26T18:22:05Z

If you were to apply a bijective function to each letter in each word of a language (e.g. English), how many pre-existing words would you obtain in the resulting image?

Since that’s a pretty convoluted way of explaining things, let’s try a more concrete example.

We’ll take the well-known

rot13

substitution cipher (a simple example of a bijection between the set of letters in the English alphabet and itself), and apply it to every letter in a chosen word. For most words, the result will be non-sensical gibberish. There does exist, however, a subset of valid English words that map into other valid English words.

Example:

rot13('sync') = 'flap'

How many of these words exist? To answer this, I wrote a small Python script that loads up the words in my system dictionary into a Redis set. Another set of the rot13′ed words is then stored, and the set intersection of the original and transformed words is calculated:



import redis
def encode(word):

    return word.encode(‘rot13′)
def cleanup():

    db.delete(‘eng’)

    db.delete(‘eng-rot13′)
if __name__ == “__main__”:

    count = 0

    db = redis.Redis()

    cleanup()
    for line in open(‘/usr/share/dict/words’, ‘r’):

        count += 1

        db.sadd(‘eng’, line)

        db.sadd(‘eng-rot13′, encode(line))

        if (count % 10000 == 0):

            print “Loaded %d words so far” % count
    db.sinterstore(‘eng-intersect’, ‘eng’, ‘eng-rot13′)

    msg = “English dictionary contains %d words, and %d rot13′ed words”

    print  msg % (db.scard(‘eng’), db.scard(‘eng-rot13′))

    print “Cardinality of intersection: %d ” % db.scard(‘eng-intersect’)

    cleanup()

After a simple cardinality check, we have our answer: 256 words^[1].

The great part about this little script is that nearly everything is done natively in Redis – the only thing Python is needed for is loading the words into the database, and the implementation of the bijective function that we wish to apply.

This is a very contrived example, but the ease with which I was able to map my thought process to code was fantastic. No need to think about tables, rows or joins – just sets, and operations on sets. The simplicity of it is almost shocking.

Pretty neat, eh?

[1] This result does not discard any single letter words (e.g. “a”), which will always trivially map into another letter when using rot13.