Nerderati

Perpetual IRC – The Multiplexer Edition

2010-09-07T15:04:35Z

As is the case with most people involved in Open Source, I’m on IRC all day long. I can help people from around the world use some of the projects I’ve helped create, as well as some of the software that I use on a daily basis.

Additionally, IRC is also the preferred form of group communication for the company I work for. One of the major advantages that IRC has over your ‘traditional’ instant messenger clients is that, with a minimum amount of effort and hardware, you can create a setup that will remain perpetually^* connected, even when you’re not online. That means that you can keep logs of conversations, receive messages, catch up on what the current topic of discussion is when starting your day, and still be able to shut down your computer at night if you so choose.

There are a few ways of achieving the status of perpetual IRC denizen. Note, however, that almost any method requires that you have access to a remote server in addition to your local machine. It doesn’t need to be anything special – a cheap VPS will do just fine, as long as you can install software packages and open the necessary ports in the firewall.

Simplest. If you want to expend the least amount of effort, at the cost of some flexibility, the easiest method of achieving IRC immortality is with the use of a terminal multiplexer, such as screen or tmux (my personal favourite), and a console-based IRC client, such as Irssi.

Don’t know what a terminal multiplexer is? Here’s a description, right from the screen homepage:

Screen is a terminal multiplexer: it takes many different running processes and manages which of them gets displayed to the user. Think of it as a window manager for your console or terminal emulator. With screen, you can have interactive processes running on your home computer and connect to them from anywhere else to check on their progress.

Let’s assume that you’ve installed screen and Irssi on your remote server. What we want to do is start Irssi inside a screen session:

$ screen -S irc irssi # name the session `irc` and start `irssi` in it

Digression. At this point, feel free to browse the documentation on how to properly setup your preferred servers, channels and other configuration parameters for the IRC client. For a quick test, you can try the following at the Irssi prompt:

/set nick YourNickName
/connect irc.freenode.net
/join #some-interesting-channel

You can also automate the process, so that upon startup Irssi will auto-connect to the servers and channels you specify:

/server add -auto -network Freenode irc.freenode.net 6667
/network add -nick YourNickName Freenode
/channel add -auto #some-interesting-channel Freenode

If you have a registered nickname, you can have it auto-identify as well:

/network add -autosendcmd "/msg nickserv identify your_pasword_here ;wait 2000" Freenode

And you’ll probably want to cut down on the noise:

/ignore #some-interesting-channel ALL -PUBLIC -ACTIONS

At this point, you should have a fully functioning IRC client inside of a screen session. You can detach from the current session by pressing control-a d, and re-attach with:

$ screen -r

See where this is going? With this setup, you can simply maintain your IRC connection within a screen session on your remote server. When you want to “log on”, you simply SSH into your remote server, screen -r, and voilà! Perpetual IRC.

This approach, while simple, has some very obvious drawbacks:

You must be on a machine that is able to SSH into your remote server.
You are confined to using the command-line IRC client running in the remote session.
Any scripts that attempt to interact with your ‘local’ desktop (e.g. Growl notifications), are painful to setup, if not impossible.

Okay, so it’s not completely ideal. But it does the job – as long as your remote screen session remains operational, you’ll be logged in to IRC.

In my next post, we’ll look at how you can achieve the same results, but without any of the aforementioned drawbacks, through the use of a specialized IRC proxy daemon.

^*: Of course, if the remote server running the terminal multiplexer session goes down, it’s not really ‘perpetual’.

Making Git Behave

2010-08-04T17:08:03Z

It’s no secret that I’m a big fan of Git, and of distributed version control in general; they offer a compelling toolset and degree of flexibility that you would be hard pressed to find in a “traditional” centralized version control system. Instead of discussing the merits of a DVCS or performing feature comparisons between particular implementations — there are enough of those already — let’s take a look at how you can bend Git to your will through configuration, and a few useful aliases. If you’ve not encountered this before, you’ll be surprised at how much can be accomplished with just a few lines in a configuration file.

Your .gitconfig

This is where the magic happens. Git, by default, will attempt to find a .gitconfig file in three places:

$GIT_DIR/config for per-repository configurations
~/.gitconfig for user-specific configurations
$(prefix)/etc/gitconfig for system-wide configurations

Read up on the various options and overrides that git-config accepts, as well as alternative methods to interactively add/remove/replace configuration options.

For the rest of this article we’ll be adding configurations to ~/.gitconfig, but there’s nothing stopping you from doing it in a per-repository (or even system-wide) configuration file instead.

For reference, here is a simplified version of the config I use:

[color]
diff = auto
status = auto
branch = auto
[user]
name = Joël Perras
email = joel.perras@gmail.com
[status]
relativePaths = false
[core]
excludesfile = /Users/joel/.gitignore
[alias]
d = diff
dc = diff --cached
lc = log ORIG_HEAD.. --stat --no-merges
gl = log --oneline --decorate --stat --graph
st = status -sb
prune-all = !git remote | xargs -n 1 git remote prune
whatis = show -s --pretty='tformat:%h (%s, %ad)' --date=short
whois = "!sh -c 'git log -i -1 --pretty=\"format:%an <%ae>\n\" --author=\"$1\"' -"

Also available as a Gist.

Now let’s go through what these configuration options actually do.

[color]

[color]
diff = auto
status = auto
branch = auto

These were, in all likelihood, the first configuration options I setup once I had installed Git for the first time. I work in a terminal window all day, and having proper and consistent colourization makes all the difference when attempting to quickly determine the state of your repository:

With these options set, all diff, status and branch commands will have (where appropriate) colourized output.

[user]

If you ever plan on making your repository available to other developers, these options are essential for identifying who you are. They’re pretty self-explanatory:

[user]
name = Joël Perras
email = joel.perras@gmail.com

[status]

The sole configuration I have under this segment is something of a personal preference. When I’m four folders deep into a repository and I invoke git status, I don’t need to see where the changed/added/deleted files are relative to where I invoked the command — I’m far more interested in their absolute location relative to the repository root. That’s where relativePaths = false comes in handy:

[status]
relativePaths = false

[core]

There are some files you just never want to commit, no matter what project you’re working on. As an avid Vim user, I sometimes have a few *.swp files lying around (mostly due to my own fault), which have no place in any project history. Instead of adding an ignore rule to each and every repository, I added this configuration directive to my ~/.gitconfig, indicating where my global never-track-any-of-these-file-patterns exists:

[core]
excludesfile = /Users/joel/.gitignore

[alias]

Now we get to the interesting stuff. Aliases in Git are exactly what you expect them to be: user-defined shortcuts that group together pre-existing git (and sometimes shell) commands and their options.

Edit: As some people have pointed out, there are a few flags/commands that will not work unless you have git 1.7.2 or above installed. Pardon the confusion.

[alias]
d = diff
dc = diff --cached
lc = log ORIG_HEAD.. --stat --no-merges
gl = log --oneline --decorate --stat --graph
st = status -sb
serve = !git daemon --reuseaddr --verbose --base-path=. --export-all ./.git
prune-all = !git remote | xargs -n 1 git remote prune
whatis = show -s --pretty='tformat:%h (%s, %ad)' --date=short
whois = "!sh -c 'git log -i -1 --pretty=\"format:%an <%ae>\n\" --author=\"$1\"' -"

Some explanations:

git d and git dc

I’m often taking diffs between different revisions in projects, so these two are quite heavily used.

git lc

Lists all new commits after you fetched. This is especially useful when working on a very active project with multiple developers.

git gl

This is the default command that I invoke when I wish to inspect the log. The output is concise as well as informative:

git st

My preferred method of viewing the status of a repository, which gets rid of the fluff that usually accompanies the help-type text of git status:

git prune-all

Since git remote prune doesn’t know how to prune all of your remotes at once, let’s teach it that trick. This is especially useful if you’re tracking many remotes from various contributors to a project where the branches get nuked/renamed on a regular basis.

git whatis <commit-ish>

Pass this command a hash, branch name or tag, and it will give you a one-liner containing the commit message associated to it as well as the date it was committed on. Useful when git show is more information than you need.

git whois <name|email>

Another one of those useful commands when you’re working on a project with a multitude of collaborators, and need to fish out an email address when all you remember is part of their name, or vice versa. Note that if the collaborator in question never configured their name and/or email address (as described above), then this command will not be very useful.

And that’s it! With nothing more than a few lines in a configuration file, we’ve managed to make Git be more informative as well as more concise.

If you have any tips/tricks/configs that you can’t live without, leave a comment below.

Quite a few of the aliases and what I would consider “sane” configuration options have been lifted from the official wiki for Git. I highly recommend perusing it, once you’ve become familiar with Git itself.

Compiling Vim With Ruby Integration On Snow Leopard

2010-08-24T13:51:17Z

One of my favourite plugins for Vim is Command-T:

an extremely fast, intuitive mechanism for opening files with a minimal number of keystrokes. It’s named “Command-T” because it is inspired by the “Go to File” window bound to Command-T in TextMate.

Sadly, the default installation of Vim on Snow Leopard does not have support for the ruby interpreter compiled in, which is a pre-requisite for using the plugin. Luckily, that’s easy enough to remedy, and in the process we’ll learn a thing or two about compiling your own custom Vim binary.

Let’s start off by getting the source code from the official Mercurial repository:

$ hg clone https://vim.googlecode.com/hg/ vim
$ cd vim

(Note: You don’t need to use the Mercurial repository – there are mirror sources for Subversion, CVS, as well as good ol’ tarballs with patches.)

The default branch for the mercurial repository contains the code for Vim 7.2 at the time of this writing. There is also a vim73 branch available for those feeling a bit more adventurous and wishing to compile the beta release of the next version. For this article, we’ll be sticking to the stable 7.2 release in the default branch.

Now, let’s take a look at the possible configuration options:

$ ./configure --help

There are quite a few, and I suggest that you take the time to read through them – most are quite self-explanatory. For Command-T, the one that we are interested in is the

--enable-rubyinterp

option.

So let’s take a shot at the simplest installation for terminal-based Vim usage, one without the GUI interface and (Linux) mouse daemon support:

$ ./configure --prefix=/my/install/prefix --enable-rubyinterp --enable-gui=no --disable-gpm
$ make

After the compilation process finishes (presumably with no errors), the first thing you’ll want to do is ensure that the binary you just built functions as expected:

$ ./src/vim --version | grep ruby
# you should see a `+ruby` line entry
$ ./src/vim

If you see the +ruby entry in the –version output and the binary launches without any errors, rejoice in your own awesomeness. That’s all there is to it.

If, however, you see something similar to this:

Vim: Caught deadly signal SEGV
Vim: Finished.
zsh: segmentation fault ./src/vim

you’ve probably fallen prey to a (currently) not very well documented issue: Vim 7.2 does not support the integration of Ruby 1.9.x on Snow Leopard.

This means that if you’ve used a package manager such as Homebrew, MacPorts or Fink (shudder) to install the latest version of Ruby, Vim will link to that latest version instead of the system default installation of ruby 1.8.7.

Let’s fix that.

We’re going to edit the src/auto/config.mk generated by configure that was run earlier. Note that if you re-run configure at a later time, your changes to config.mk will be lost.

Find the lines that look like this:

RUBY = /usr/local/bin/ruby
RUBY_SRC = if_ruby.c
RUBY_OBJ = objects/if_ruby.o
RUBY_PRO = if_ruby.pro
RUBY_CFLAGS = -I/usr/local/Cellar/ruby/1.9.1-p378/include/ruby-1.9.1 -I/usr/local/Cellar/ruby/1.9.1-p378/include/ruby-1.9.1/i386-darwin10.4.0 -DRUBY_VERSION=19
RUBY_LIBS = -lruby -lpthread -ldl -lobjc

(Note: Your specific paths and/or versions may differ depending on the package manager that you are using. The above paths are actually not important, however, since we actually want to reset them to the system defaults.)

and replace them with the following:

RUBY = /usr/bin/ruby
RUBY_SRC = if_ruby.c
RUBY_OBJ = objects/if_ruby.o
RUBY_PRO = if_ruby.pro
RUBY_CFLAGS = -I/System/Library/Frameworks/Ruby.framework/Versions/1.8 -I/usr/lib/ruby/1.8/universal-darwin10.0
RUBY_LIBS = -framework Ruby

Alright, let’s see if this worked.

$ make clean && make

Before we check the binary as we did before, let’s see if we linked to the correct ruby libraries:

$ otool -L src/vim
src/vim:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.0)
/usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/libruby.1.dylib (compatibility version 1.8.0, current version 1.8.7)
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 44.0.0)
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 550.29.0)

Looking good so far – the binary is linked to the framework version of Ruby that comes as a default on Snow Leopard.

Let’s do a version check:

$ ./src/vim --version | grep ruby
# you should see a `+ruby` line entry
$ ./src/vim

And voilà: A custom-built Vim with ruby integration that will happily run the Command-T plugin.

All that’s left is to install it:

$ make install

Assuming your PATH is setup to correctly find the new Vim binary, you should be all set.

WebNotWar

2010-05-20T21:41:30Z

I’m quite happy to announce that I will be giving the keynote address at this year’s WebNotWar/For The Web conference, taking place on May 27th, 2010.

First, it’s a free conference for attendees. Being an open-source kinda guy, I know & love the word free. Second, the theme of the conference is ‘interoperability’, which is something very important to me, to my open source work, to my job, and to end users (I like to call them ‘normies’) – so much so that my keynote will address this directly.

The technological landscape of the web is diverse, and keeping track of new developments and projects is a full time job. Luckily, conferences like this one provide a great way to meet the people that make the web of tomorrow, and get a head start on what the next Big Thing is going to be.

I hope to see you there!

The Wonders and Simplicity of Redis Sets

2010-02-26T18:22:05Z

If you were to apply a bijective function to each letter in each word of a language (e.g. English), how many pre-existing words would you obtain in the resulting image?

Since that’s a pretty convoluted way of explaining things, let’s try a more concrete example.

We’ll take the well-known

rot13

substitution cipher (a simple example of a bijection between the set of letters in the English alphabet and itself), and apply it to every letter in a chosen word. For most words, the result will be non-sensical gibberish. There does exist, however, a subset of valid English words that map into other valid English words.

Example:

rot13('sync') = 'flap'

How many of these words exist? To answer this, I wrote a small Python script that loads up the words in my system dictionary into a Redis set. Another set of the rot13′ed words is then stored, and the set intersection of the original and transformed words is calculated:



import redis
def encode(word):

    return word.encode(‘rot13′)
def cleanup():

    db.delete(‘eng’)

    db.delete(‘eng-rot13′)
if __name__ == “__main__”:

    count = 0

    db = redis.Redis()

    cleanup()
    for line in open(‘/usr/share/dict/words’, ‘r’):

        count += 1

        db.sadd(‘eng’, line)

        db.sadd(‘eng-rot13′, encode(line))

        if (count % 10000 == 0):

            print “Loaded %d words so far” % count
    db.sinterstore(‘eng-intersect’, ‘eng’, ‘eng-rot13′)

    msg = “English dictionary contains %d words, and %d rot13′ed words”

    print  msg % (db.scard(‘eng’), db.scard(‘eng-rot13′))

    print “Cardinality of intersection: %d ” % db.scard(‘eng-intersect’)

    cleanup()

After a simple cardinality check, we have our answer: 256 words^[1].

The great part about this little script is that nearly everything is done natively in Redis – the only thing Python is needed for is loading the words into the database, and the implementation of the bijective function that we wish to apply.

This is a very contrived example, but the ease with which I was able to map my thought process to code was fantastic. No need to think about tables, rows or joins – just sets, and operations on sets. The simplicity of it is almost shocking.

Pretty neat, eh?

[1] This result does not discard any single letter words (e.g. “a”), which will always trivially map into another letter when using rot13.

Shebang

2010-01-31T07:04:45Z

An explanation of the shebang^[1], and what it means when included in a script:

#!/path/to/interpreter -flags

Means

“This file is not My Words, but My Commandment to you, System. You must travel along this Path. At the end, you will find an Interpreter. You will pass unto him these Flags, and he will help you to understand My Biddings. You will do this, for I have execution permissions on this file.”

From the bowels of Reddit.

[1]: Shebang (#!)

Macro Humanity

2010-01-12T01:18:42Z

It’s not often that I come across an essayist that completely blows me away, but Nick Bostrom has done just that.

I came across his name for the first time while browsing the latest Hacker News submissions, under the link-baited submission title: Why I Hope the Search for Extraterrestrial Life Finds Nothing. Needless to say, I was intrigued, and the comments on the HN page were encouraging.

After a few cursory credential checks¹ to confirm that the author wasn’t a complete nut job (quite the opposite, actually – see the footnote), I took the time to sit down and read the essay.

Here is a short excerpt:

What could be more fascinating than discovering life that had evolved entirely independently of life here on Earth? Many people would [...] find it heartening to learn that we are not entirely alone in this vast cold cosmos.
But I hope that our Mars probes will discover nothing. It would be good news if we find Mars to be completely sterile. Dead rocks and lifeless sands would lift my spirit.

Where Are They? [pdf]

I won’t go into any of the finer details – for that, you should read Professor Bostrom’s elegantly written article – but If you’ve ever heard of the Fermi Paradox or pondered over the murky details of the Anthropic Principle, I highly recommend reading the article I’ve linked to above. These topics may have been discussed ad nauseam in many corners of the web, Professor Bostrom’s thoughts and insights are trully a breath of fresh air.

[1] Professor Bostrom has quite the resumé, including a Wikipedia page, two published books, dozens of published articles in well respected journals, and is currently the director of The Future of Humanity Institute at Oxford University. In other news, I now feel more inadequate than ever.

Redis Memory Monitoring – Python Edition

2010-01-05T04:58:05Z

A few hours ago, Salvatore Sanfilippo (the lead developer of Redis), tweeted a little Ruby script to interactively estimate the memory usage of a running

redis-server

instance:



require ‘rubygems’

require ‘redis’
def main(opts={})

    r = Redis.new(opts)

    um = 0

    while true do

        newum = r.info[:used_memory]

        if newum != um && um != 0

            diff = newum.to_i-um.to_i

            puts “#{um} bytes (#{diff} difference)”

        end

        um = newum

        sleep 1

    end

end
main

I’ve been experimenting with a Python+Redis combination (with redis-py) for data analysis on a few side projects lately, and a simple script like this can come in handy when you want to make sure you’re not doing something completely stupid with Redis that gobbles up all of the allocated memory. And yes, I’ve been guilty of doing that on a few occasions.

Converting the script from Ruby to Python (with some additional logic for command line option parsing) is very straightforward:



#!/usr/bin/env python
import redis

from optparse import OptionParser

from time import sleep
def options():

    parser = OptionParser()

    parser.add_option(“–host”, default=”localhost”)

    parser.add_option(“–port”, type=”int”, default=6379)

    return parser.parse_args()
if __name__ == ‘__main__’:

    (opts, args) = options()

    r = redis.Redis(host=opts.host, port=opts.port)

    um = 0
    while True:

        newum = r.info()['used_memory']

        if newum != um and um != 0:

            print(‘%d bytes (%d difference)’) % (um, newum – um)

        um = newum

        sleep(1)

Once again, git and GitHub make this kind of collaborative development almost too easy.

ConFoo You Too

2009-12-15T07:11:19Z

While a bit late, I’m extremely happy to announce that I have been selected as a speaker for the ConFoo.ca Conference to be held in Montréal at the beginning of March, 2010.

I attended this conference last year when it known as PHPQuébec, and had a fantastic time; the sessions as well as the speakers were excellent, as were the hallway conversations with other conference attendees.

Along with the change in name, the focus of the conference itself has shifted from being PHP-centric to something more language and technology agnostic, with sessions on .NET development, Python tricks & idioms, databases and a host of other topics. The list of sessions and speakers should make any developer worth his salt giddy with anticipation.

The talk I will be presenting is entitled A Web Framework for People Who Hate Frameworks, and focuses on Lithium, one of my most recent endeavours with CakePHP core alumni @nateabele and @gwoo. You can read more about the talk I’ll be giving on the session page, and for all those who will not be attending ConFoo 2010 I encourage you to visit

#li3

#li3-core

on irc.freenode.net, and ask us why Lithium is making waves in the web framework world.