I'm a lumberjaph

Practical Joke

2013-02-24T00:00:00+00:00

I’ve nothing exciting to write about so I’ll share a prank I did a few years ago (because I’m kinda proud of this one).

To protect the innocent, I’ll change all the names.

I was working at a company named $LF. One of my colleague ($NG), always looks at his keyboard when he types. Some time he can type for a few minutes. Then he looks at his screen, only to realize that it’s full of typo, or worse, that he had typed in the wrong window. This used to really bother me, because I know he’s been typing for years, but I was not sure if he was looking at his keyboard because he needs to, or just because his head is too heavy to look at the screen.

So one day, while everybody (except $GM) were away taking a break, I decided to conduct an experiment.

The french keyboard uses the AZERTY layout. $NG uses the best editor in the world, where the x key is really important (try to do something without ‘Meta-x’). He also uses a laptop, connected to an external monitor and with an external keyboard. So I decided to swap the x and the w letter on the external keyboard, by switching the caps. The actual x and w were still at the same place, only the label on the key was switched.

When he came back from the break, he starts typing as usual. Quickly he realized that something was wrong, and he did a lot of things to figure out what. Among them:

verify that the content of xmodmaprc was correct, and then tweak it
reinitialize the layout with setxkbmap
use a tool (maybe showkey ?) to see what was the actual code send from the keyboard while hitting the key
probably recompiled a bunch of stuff
definitely upgraded his Linux distribution

At this point, he understood that when hitting x, w was received, and for w, x. He rebooted his laptop (because when in doubt, you should “tried turning it off and on again”). But the problem was still there.

If I remember correctly, that’s when he decided to look at the laptop’s keyboard and tried to use it, only to realize that it was working correctly! So he asked the ops guy, $GM, who was sitting next to me, if there was a spare keyboard somewhere. I slowly turned my head toward $GM, whispering ‘noooo’ to him, and he replied to $NG “nop, sorry, no spare keyboard, <insert stupid excuse #49>”.

The other developers started to realize that something was wrong with $NG, but no one said or did anything yet. We were about 30 minutes in the experiment now.

Because he still had work to do, he tried to cop with the problem, and every time he needed a x or a w, he would type something, stop, look at the screen, sigh/grumble, and use the laptop’s keyboard to fix all the w and x. That was difficult to miss, and soon other people asked him what was wrong, and so that’s when I revealed .

When I explained what I did he was surprised, starred at the two keyboards for a while, and saw that, effectively, the keys were swapped! I’m pretty sure the rest of the team made fun of him but I don’t really remember exactly how.

I consider the experiment to be a success: I proved that he need to look at his keyboard to type, and I’m also confident that I can reproduce it with the same subject multiple time and obtain the same result again.

Should I read the code ?

2013-02-19T00:00:00+00:00

This conversation happened twice in the last few weeks at work, the first time during my 1:1 with my manager, and a second time with the whole team.

We were investigating Riemann, and we started to discuss what it would means to adopt this technology in our stack. Riemann is written in Clojure, and no one at work is really familiar with this language (except for me, and I’m don’t consider myself efficient with it).

The question is how do you deal with a new tool when there’s only one person in the team that can read the code, and therefore contribute to the project to add features, fix bugs, etc. Are we supposed to be familiar with the code of the things that we use?

I’ve never read the code of MySQL, I’ve read parts of Apache’s code, and I’ve never looked at the source of the Linux’ kernel. At the same time, I usually read the code of the Perl and Python libraries I use frequently, I’ve also read the source of statsd and Graphite (two other tools that we looked at) in order to understand what they do and hunt for issues (in the way we use them) or bugs.

I see two ways to approach this question so far: as a developer and as a user. As a developer, I consider that I have to read and understand the code of the libraries my code depends on (we’ve found some serious issues in libraries we use daily because of this approach).

For services we use in the infrastructure, it depends of the size of the tool and it’s community. For a new product, or when the documentation is too sparse, or when the community is rather small, it’s a good thing to be able to look at the code and explain what it does. For bigger projects (MySQL, Apache, Riak), so far I’ve relied on the experiences people had with the tools, the community.

I’ll conclude this post with an anecdote. Last Thursday we were trying to understand why the CPU load on the Graphite’s box went over the roof when we added about 25% more metrics to it. With Abe and Hachi we said “ok let’s dig into this problem”. You could have guessed who are the ops while looking at the scene. We were looking for the same things: reads and write. Abe and Hachi started to do that with the help of strace, while I started to walk through the code. I think the two ways are valid, at least you can use one to correlate the other, and they gave you different information (strace will help you to time the operations, while the code would explain what you’re writing and reading).

I’m curious to hear how other approach this problem.

Let's talk about Graphite

2013-01-28T00:00:00+00:00

As I’ve already mentionned in a previous post, at $work we are currently deploying Graphite and the usual suspects.

Finding articles on how to install all these tools is easy, there’s plenty of them. But what’s really hard to find, are stories on how to use them: what’s collected, how, why, how do you organize your metrics, do you rewrite them, etc.

What I want with this post is to start a discussion about usages, patterns, and good practices. I’m going to share how we, at SAY, are using them, and maybe this could be the start of a conversation.

Graphite/collectd/statsd

We’re using statsd, collectd Graphite (and soon Riemann for alerting).

Retention

Our default retention policy is: 10s:1h, 1m:7d, 15m:30d, 1h:2y. We don’t believe that Graphite should be used for alerting: it’s a tool for looking at history and trends.

360 points for the last hour is enough to refer to a graph when an incident occurs. Most of the teams are releasing at least once a week, so 1 minute definition for a week is enough to compare trends between two releases. Then we go to a month (2 sprints) and they 2 years. We thought at first to keep only 15 months (1year + 1 quarter to compare), but since we have enough disk space, we decided to keep two years, however we might decide to change that in the future.

System

We don’t do anything fancy here. collectd is running on each host, and then write to a central collectd server.

Services

For shared services (Memcached, Varnish, Apache, etc), we talk to statsd. We have a Perl script named sigmoid, with the following usage:

Usage: ./sigmoid [<options>] <metricname> <value>
 Exactly one of:
 --counter (value defaults to 1)
 --aggregate
 --gauge
 --event
 --raw

 Other options:
 --disable-multiplex (for statsd only)
 --appname
 --hostname (to log on behalf of another machine)

This script is used by other scripts who monitor logs, status of apps, etc. This way it’s very easy for a Perl, Python, Shell script to just call sigmoid via system, and then send the metric and the value to statsd.

For some other services we might need something more specific. Let’s take a look at Apache. We have another Perl script for the CustomLog settings (CustomLog "|/usr/local/bin/apache-statsd"). The script is doing the following things:

compute the size of the HTTP request in bytes
compute how long it took to return the response

Then, it will send the following lines to statsd (with $base being the vhost in our case):

$base.all.requests:1|c increases the total of HTTP requests we’re receiving
$base.all.bytes:$bytes|ms send the size, in bytes, of that request
$base.all.time:$msec|ms the time spend to get the response

Now we will send the same line two more times, with a different prefix: $base.method.$request_method and $base.status.$status.

Applications

Here, developers decide what they want to collect, and send the metric to statsd.

Events

And finally we have events. Every time we push an application or a configuration, we create a new event.

Proxying statsd

We want metrics to be well organized, in a clear hierarchy. Jason Dixon wrote in a blog post that Misaligned paths are ok. I disagree. We’re collecting more than 100k metrics so far. If things are not well organized, it will become quickly very difficult to find what you’re looking for.

So, here’s how we organize our metrics. The first level is the environment (PROD, CI, DEV, …). Then we have apps and hosts. For the host section, we group by cluster type (Hadoop cluster, Web servers for TypePad, etc), and then you have the actual host, with all the metrics collected. For apps, we have four main categories: aggregate, counters, events and gauges (I’ll come back on that later).

Earlier I said that apps where sending metrics to statsd, but that’s not exactly true. We (mostly) never write directly to statsd or Graphite.

On each host, we have a Perl script listening. This proxy will rewrite all the incoming metrics by appending to the name the environment, the cluster and so on. This way when someone want to send a key, he doesn’t have to care convention or using the correct prefix.

Also, it will also multiplex the metric: we want the same key to end-up under host and under app. Let’s take an example here. If you’re writing a web service, you may want to send a metric for the total time taken by an endpoint (this will be an aggregate). Our key will be something like: <application-name>.<endpoint-name>.<http-method>.<total-time>. The proxy, based on the network address, will determine that it’s environment is CI, and that it’s an application. But it also knows the name of the server, and the cluster. So two keys will be created:

<CI>.<apps>.<aggregate>.<application-name>.<endpoint-name>.<http-method>.<total-time>
<CI>.<hosts>.<cluster-name>.<host-name>.<aggregate>.<application-name>.<endpoint-name>.<http-method>.<total-time>

This way we can find the metric aggregated by application, or if we think there’s a problem in one machine, we can compare per host the same metric.

Meetup

We’ve a huge basement at work that can be used to host meetup. There’s already a few meetup in the San Francisco about “devops” stuff (Metrics Meetup, SF DevOps, etc), but maybe there’s room for another one with a different format.

What I would like, is a kind of forum, where a topic is picked, and people share their experiences (the bad, the good and the ugly), not how to configure or deploy something. And there’s a lot of topics where I’ve questions: deployment (this will be the topic of my next entry I think), monitoring, alerting, post-mortem, etc. If you’re interested, send me an email, or drop a comment on this post.

Where I'm excited about a keyboard

2013-01-14T00:00:00+00:00

One thing that I always find fascinating is how an object, or a way of thinking, was modeled 25, 50 or 100 years ago, and how we keep going on with the same design without trying to think if it’s actually the right one, or to challenge our way of thinking with a different one. Keyboards are one of those things. We all know the origin of the QWERTY layout, why the keyboard has this shape, and so on (the wikipedia page is a good read). Let’s take a look at this tweet from Technomancy (who, for the the record, also has some interesting keyboard ideas :)

I wonder how long it'll be before we can look back and laugh at the idea of poking a QWERTY soft-keyboard on a touch screen. Hope it's soon.
— Phil Hagelberg (@technomancy) January 14, 2013

That’s exactly what I’m talking about: we keep reproducing the same design without questioning it’s benefits (I’ve no doubt that people at Apple and Google have tried a lot of alternatives, an they probably decided to stick to this one because it’s the most common and people are used to it. Still, it doesn’t mean we have to stick with it). But even for physical keyboard, we keep the same design: a large space key (why ?!), a capslock key (who need that one, even remapped to “control” I feel like it’s a bad place to physically put a key on the keyboard), large enter/shift keys, etc.

I’m not a keyboard nerd. But sometimes I take a look at some new and strange models, sometimes I try one of them, and I might even buy one, like the TypeMatrix (that one I really liked).

So, when Jesse announced that he was working on a keyboard, I got really excited. His layout is very interesting (it seems to be inspired by the kinesis), and the shape is also innovative. I also remember reading that he suffers from RSI, which mean it’s something he will be careful bout. Take a look at the mapping: there’s only 52 keys, control/shift/alt are repeated, but with this shape it makes sense, since they’re dedicated to one half of the keyboard.

You can read about his progress. He’s also publishing pictures on his flickr account.

I really hope he will be able to build this keyboard and manage to manufacture it (maybe a kickstarter project ?). I’ll definitely be in line to get one.

Carbon's manhole

2013-01-10T00:00:00+00:00

We’re rolling out Graphite and statsd at work, and I’ve spend some time debugging our setup. Most of the time, the only thing I need is tcpdump to verify that a host is sending correctly the various metrics.

But today, thanks to a stupid reason, I’ve learned about another way to debug carbon: the manhole. The idea of the manhole is to give you a access to a REPL attached to the live process. When my boss told me about it, I was at first surprised to see this in a Python application. I’ve already been exposed to this kind of debugging thanks to Clojure, where it’s not uncommon to connect a REPL to your live application (for example, Heroku document how to connect to a remote live REPL in your application). When I first heard of that I was very skeptical (give access to a live environment, and let the developer mess with the process ?!). But I’ve learned to love it and I feel naked when I’m working in an environment where this is not available. So I was happy to jump and take a look at that feature.

Since it’s not very well documented and I had a hard time finding some information, let me share here the basics.

First you’ll need to configure Carbon’s to allow the connection:

ENABLE_MANHOLE = True # by default it's set to False
MANHOLE_INTERFACE = 127.0.0.1
MANHOLE_PORT = 7222
MANHOLE_USER = admin
MANHOLE_PUBLIC_KEY = <your public SSH key, the string, not the path to the key>

Now you can restart carbon, and connect to the Python shell with ssh admin@127.0.0.1 -p7222. This manhole is useful to get an idea of the data structure your process is handling, or to get an idea of what’s going on (is there a lot of keys being held in memory? Is the queue size for one metric huge? etc).

From here, you can execute Python code to examine the data of the process:

>>> from carbon.cache import MetricCache
>>> print MetricCache['PROD.apps.xxx.yyy.zzz]
[(1357861603.0, 93800.0), (1357861613.0, 98200.0), (1357861623.0, 91900.0)]

The MetricCache class is a Python dictionary where you can access your keys. You can also list all the metrics with the size of their queue with MetricCache.counts().

Or even force the daemon to write to disk all the data points:

>>> from carbon.writer import writeCachedDataPoints
>>> writeCachedDataPoints()

Before doing any of that, I would recommend to read the code of carbon. It’s pretty short and quiet straight forward, especially the code of the writer.

Of course, you have to know what you’re doing when you’re executing code from a REPL in a live environment.

About DevOps

2012-12-16T00:00:00+00:00

There’s a lot of talk about what is or what is not DevOps, and I’ll throw my opinion in the mix.

Until a few weeks ago, at work, we had mostly two teams: the engineering team and the ops team. Our workflow was (to simplify) the following:

engineers develop services and applications
they push their change to Jenkins
a build pass and is pushed to CI
a few times a week, engineers ask Ops to push the change to production

There’s already a lot of articles about the kind of frictions created by this (who owns what; engineers would blame ops when the push was failing (or the other way around); it’s hard for ops to know what’s wrong when something is broken in production; etc).

A few weeks ago it was decided to create a new team to improve engineers efficiency, and the team was named “DevOps”. At first I was not sure it was the right name for this team, but now I don’t think it matters.

I was not sure the name was appropriate because of this article (and a few other to respond), explaining why you don’t want a DevOps team, but instead you want the whole organization to be DevOps. We need engineers to own their applications, to be able to push when they want, but also to monitor, know what’s wrong or slow, etc.

The work hachi and I will have to do is to help engineers and ops to be the DevOps. Our team responsibility is to choose, evaluate and integrate tools. We will also provide libraries, documentation, training and support. We are not the DevOps. Our goals are to create this culture, to give more responsibilities to engineers, and to free Ops from the work of pushing code. The success of this team will be measured by the adoption of our work by engineers and ops.

So yes, I agree that you don’t want a dedicated DevOps team, but you still need a team coming from different background (hachi is coming from Ops and I’m an engineer) to build that culture.

Perl, Redis and AnyEvent at Craiglist

2012-11-28T00:00:00+00:00

Last night I went to the SF.pm meetup, hosted by Craiglist (thanks for the food!), where Jeremy Zawodny talked about Redis and his module AnyEvent::Redis::Federated. We were about 30 mongers.

I was eating at the same table as Craiglist CTO’s, and he went through some details of their infrastructure. I was surprised by the quantity of place where they use Perl, and the amount of traffic they deal with.

Redis

Jeremey started his talk by explaining what is their current problem: they have hundred of hosts in multiple data center, and they collect continuously dozen of metrics. They looked at MySQL to store them, but it was too slow to support the writes. Another thing important for them is that mostly only the most recent data matters. They want to know what’s going on now, they don’t really care about the past.

So their goal is simple: they need something fast, really fast, and simple. That’s where Redis enter the game.

They want data replication, but Redis don’t have this feature: there’s only a master/slave replication mechanism (so, one way), and they need a solution with multi master, where a node becoming master does not drop data. They address this issue with a “syncer”, that I’ll describe later.

Because Redis is single thread, and servers have multiple cores, they start 8 process on each node to take advantages of them.

To me, the main benefit of Redis over Memcached is that you can use it as a data structure server. If you only need something to store key value, I’ll prefer to stick to memcached: the community around is bigger, there’s a lot of well know patterns, and a lot of big companies are contributing to it (lately, Twitter and FaceBook).

The structure they use the most are the sorted set. The format to store a metric is:

key: $time_period:$host:$metric (where the $timeperiod is usually a day)
score: $timestamp
value: $timestamp:$value

In addition of storing those metrics in the nodes, they also keep a journal of what has changed. The journal looks like this:

score: $timestamp of the last time something has changed
value: $key that changed

The journal is only one big structure, and it’s used by their syncer (more about that in a moment). The benefit of having ZSET is that they can delete old data easily by using the key (they don’t have enough memory to store more than a couple of days, so they need to be able to delete by day kickly).

The journal is use for replication. Each process has a syncer that track all his peers, pull the data from those nodes and merge them with the local data. Earlier Jeremy mentioned that they have 8 instances on each node, so a the syncer from process 1 on node a will only check for the process 1 on node b.

He also mentioned a memory optimization done by Redis (you can read more about that here).

AnyEvent::Redis::Federated

Now, it’s time to see the Perl code. AnyEventE::Redis::Federated is a layer on top of AnyEvent::Redis that implements a consistent hashing. I guess now every body has gave up hope to see someday redis cluster (and I’m more and more convinced that hit should never be implemented, and let the client implement their own solution for hashing / replication).

Some of the nice feature of the modules:

call chaining
you can get singleton object for the connection
you can also use it in blocking mode
query all node (where you send the same command to all the node, can be useful to do sanity check on the data)
the client will write to one node, and let the syncer do the job

He then showed us some code (with a very gross example: new AnyEvent::Redis::Federated, I know at least one person who would have probably said something :).

Concerns

The idea seems fine, but, as one person noted during the Q&A, how will this scale when you have more than 2 or 4 nodes in your cluster ? Since each process’ syncer need to talk to all the other nodes, it will probably be very expensive for this process to gather information from all the nodes and write them. Also, by adding more nodes, you’re storing less information into each process, since you replicate everything. Maybe a good solution is to keep many small cluster of 2 to 4 nodes, and let each of them deal with some specific metrics.

The module is not yet used in production, but they’ve tested it heavily, in a lot of conditions (but I would note that there’s no unit test :). They intent to use it soon with some home made dashboard to display the metrics.

Where we talk about ansible and chef

2012-11-27T00:00:00+00:00

I’ve been using Chef for some time now, but it was always via Vagrant, so a few weeks ago I decided to get more familiar with it. A friend of mine had set up a Chef server for his own use and was OK to let me use it for my personal server. There was a four days weekend for Thanksgiving coming, so it was the perfect occasion to take a better look at it, and to re-install my Linode server with Chef. And since it was a really long weekend, I also decided to take a look at ansible, another tool to push stuff to your server.

I’m not going to talk about installation, configuration, set up and all that kind of stuff, there’s enough material available around (blog posts, articles, books, etc). Instead, I will talk about my experience and share my (very valuable) opinion (because, clearly, the world deserve to know what I think).

Writing cookbooks for Chef

For the few of you who don’t know, cookbooks, in Chef’s world, are a group of files (templates, static files) and code that are used to automate your infrastructure. Usually, you’ll create a cookbook for each of your application (one for nginx, one for MySQL, etc).

I’ve a few services on my server (git, gitolite, Jenkins, graphite, collectd, phabricator, …), and I wanted a coobook for each of them. I’ve started by looking for the one already existing (there’s a lot of them on GitHub, on the opscode’s account), and I tried to use them without any modification. Usually, a cookbook will let you set some configuration details in your role or node to override the defaults it provides (like the password for MySQL, or the path where to put logs). So what I did was to set the interesting cookbook as a git submodule in my cookbook repository. Unfortunately, for almost all of them, I had to give up and import them in the repo, so I could edit and modify them.

That’s probably my biggest complaint with cookbooks: I doubt code re-usability is possible. You can use a cookbook as a base for your own version, but either they are too generic; or sometimes you need a workaround; or they do way too many things. And as a result, you need to edit the code to make them behave the way you want.

In my opinion, developers/ops should just publish LWRP (Lightweight Resources and Providers) and templates, that’s the only thing that I can see as really re-usable (take a look at perl-chef, I think that this one is a good example).

Using ansible

ansible was a new tool for me. A few friends mentionned it to me last October when I was at the OSDC.fr and it was also suggested to me by a colleague at work.

This tool is definitely less known that Chef, so I’ll give a quick introduction. In ansible world, you write “playbooks”, which are the orchestration language for the tool. That sounds very similar with Chef, but the main difference is they are not actual code, but a scenario with actions.

On the web site of the project, there’s a quote saying:

You can get started in minutes.

and for once, that’s true. I only had to read the first page of the documentation, and I was able to write a very simple playbook that I was able to evolve very quickly to do something actually useful.

Another difference with Chef is that they don’t incite you to share your playbooks, but instead to share your modules. Modules could be compared to Chef’s LWRP. They are Python code to do something specific (like the pip module, to install Python package, or the template’s one).

Chef vs Ansible

For now, I’ve decided to stick to this: use Chef for my supporting application (nginx, MySQL, etc) and ansible for my own applications.

So far, I prefer ansible to Chef. There’s definitely less available material about ansible on the net, but the quality is better, and the main documentation is very (I insist on the very) well organized. I’ve never spend more than 10 minutes looking for something and to implement it. I can’t say the same with Chef: the wiki is confusing; there’s way too many cookbooks available; their quality is very disparate.

Two tech. talks in a day

2012-11-14T00:00:00+00:00

Today I assisted to two tech. talks. One of them was our “Reading Group” session at lunch and the second one was the Python meetup tonight.

SAY’s tech talk

I’m trying to organize at work, every two weeks during lunch time, a session where engineers can discuss about an article, tool, or paper they find interesting. Today we were a very small group (only 4 peoples), and we talked about two tools that Masa wanted to explore: Kage and HTTP Archive.

Kage

We talked about Kage first (kah-geh means “shadow” in Japanese). Masa started to explain what was the goal of the tool and how it works. Basically, it’s a proxy that will send your HTTP request to your production and stage environment. The basic example show how easy it is to write a simple proxy with a few rules to dispatch only your GET request to your two environments, and then compare the payload returned.

It’s using em-proxy which is a DSL to write proxy for EventMachine. We then discussed where we could use it in our architecture and what benefit we could get from it, but for now we don’t have any plan to use it.

HAR

Then the second thing we talked about what HAR. You can find more detailed information on Ilya’s blog and on this gist. To summarize, HAR is an HTTP archive format. When you use the developers tools in Chrome, you can see the request and their response time, but you can also save this information in the HAR format.

There’s a few tools then to analyse this data, like YSlow, or browse them with the HARviewer. Another interesting thing is that you don’t have to use a browser to get them, you can use Phantomjs. Masa and Kyle talked about it and if they could use that with Nagios, or with Jenkins to measure the response time.

SF Python Meetup

The last talk of the day was the SF Python Meetup after a 4 or 5 months break, at the Yelp headquarter. We were supposed to have some lightning talks to start, but we only had one, from someone who think that “lightning talk” means “publicity for Google and ho by the way, we’re hiring” (yeah like no one in the room knew that …).

Then David Schachter presented How to Speed Up A Python Program 114,000 times. He showed us how he improved the performance of a script by 114,000 times. I will start by saying the talk was interesting and that David was an entertaining speaker. He went through some optimization he used, like using multiprocessing modules, cython, and some more difficult optimization he was not able to get from cython (like the permuted vector code).

He had a very strong opinion about cluster and I really disagree with him. One of his complains was that cluster are hard, are young, and we don’t have any tools, so we should not use them. But we’ve been using cluster in universities and laboratories for years now. Even if the tools are still not very great, they exist, and they work. And more importantly to me, the fact that he spend 12 weeks on optimizing his program, and doing stuff that I would not be able to do, he proved that optimizing is also very hard, and that doesn’t seem easier to me that using a cluster.

All in all, that was a good day, and I learned new things. Now I need to find a subject for our next reading group session.

Virtualenv and checkouts

2012-10-31T00:00:00+00:00

I’ve started to do some Clojure in my spare time. The default tool adopted by the community to manage projects is leiningen. For those of you who don’t know what lein is, it’s a tool to automate your Clojure project: it will boostrap a new project, install the dependencies, and there’s a plugin mechanism to extend the default possibilities of the tool.

One of the nice feature of the tool is the checkouts/ directory. From the FAQ :

If you create a directory named checkouts in your project root and symlink some other project roots into it, Leiningen will allow you to hack on them in parallel.

For Python projects at $work I use virtualenvwrapper to easily work on them without having to deal with conflicting dependencies. When I need to change a library that is used by one of the project, usually I go to the virtualenv directory and create a symlink so it uses the one I’m editing.

What I really want is a mechanism similar to lein, where I can have a checkouts/ directory inside the main project, where I can clone a library or create a symlink. Since virtualenvwrapper provides a hook mechanism, I wrote a small hook inside ~/.virtualenvs/postactivate:

#!/bin/bash

# move to the directory of the project
proj_name=$(echo $VIRTUAL_ENV|awk -F'/' '{print $NF}')
proj_path=/home/vagrant/src/$proj_name

cd $proj_path

if [ -d checkouts ]; then
    for ext in $(ls checkouts); do
        export PYTHONPATH=proj_path/checkouts/$ext:$PYTHONPATH
    done
fi

Then, when I type workon $project_name in my shell, the environment is activated, I’m moved to the right directory, and the library inside the checkouts/ directory are added to my PYTHONPATH.