One of the keys to project man­age­ment is know­ing and embrac­ing the lim­i­ta­tions of a project. One of the other keys is being able to help others see and embrace the same limitations.

In soft­ware devel­op­ment there’s been a lot of talk about “Fast. Cheap. Good. Pick two”. But I think the most common iron tri­an­gle we face is actu­ally “Scope. Sched­ule. Qual­ity. Pick two.”

Very few clients would ever actu­ally pick Scope and Sched­ule, and say “Meh. To hell with quality.”

So the real lim­i­ta­tions of our dis­ci­pline are Scope and Sched­ule, because almost no one wants an on-​time piece of garbage, or an incom­plete piece of garbage.

There are two strate­gies that I use to embrace these limitations:

Kanban (fixed scope)

If you haven’t heard of Kanban, I rec­om­mend you read up on it as it really is a great system. Here’s just one quick tuto­r­ial, and my run­ning list of links.

For me, the essence of Kanban is fixed scope, but flex­i­ble sched­ule. Sure, you can quote a lead time — an aver­age of how long it’s taken fea­tures to go from the back­log all the way to done in the past — but that’s an esti­mate, not a commitment.

This works great if you’ve got clients who are flex­i­ble enough to accept a fairly open-​ended deliv­ery date.

Scrumban (fixed schedule)

Scrum­ban is a great tran­si­tional step for teams and clients when trying to go from Scrum/Agile or, shud­der Water­fall, to Kanban.

In Scrum­ban you’ll fix a sched­ule, a release every two or three weeks sim­i­lar to Scrum. But unlike Scrum you don’t fix the scope for the sprint.

Instead you work with the client and the team to come up with a pri­or­i­tized back­log of items for that sprint.

Work during the sprint is popped off the top of the back­log stack, so that the most impor­tant work (as judged by the client) is done first.

If the back­log stack gets too small, then, aside from cel­e­brat­ing your pro­duc­tiv­ity, the team and client can regroup and fill the back­log up again.

If the back­log isn’t com­pleted by the end of the sprint, that’s OK because we decided to fix sched­ule (and qual­ity), not scope.

In con­sul­ta­tion with the team and the client, you can take the remain­ing items and start the next sprint with them, or throw them all out if pri­or­i­ties have changed!

If you can find one who thinks it’s a good thing, then you’ve got your­self a bad system admin­is­tra­tor. A good sysad­min knows that if a box is pegged at 100 per­cent, then it can’t handle any more traf­fic and it’s likely to fall over at any moment.

Show some man­agers an employee work­ing at 100 per­cent. Many will say, while pat­ting them­selves on the back, that must be a great worker who’s been prop­erly motived by a good manager… wink.

But some, myself included, would think “there’s an employee who can’t handle any more traf­fic and is likely to fall over at any moment.“

Machines can’t run at 100 per­cent all the time, and nei­ther can humans. That’s why we have slack time.

Slack time ben­e­fits employ­ees by giving them time to cool down and recharge their cre­ativ­ity between projects. Man­agers ben­e­fit by con­stantly having a pool of refreshed, ready talent to han­dling incom­ing work.

Good man­agers should have a pool of low pri­or­ity, deadline-​less, “it’d be nice if” projects and tasks.

That way, when higher pri­or­ity work has been com­pleted and there’s down time, there’s some­thing for folks to work on that can be easily set aside should an emer­gency arise, or new work arrive.

Editor’s note: This is the first in a series of posts where I’ll try and espouse on the some of the prin­ci­pals and prac­tices of Kanban.

1. Stop the repetitive announcements

How many times do we need to announce over the PA system that “the white zone is for load­ing and unloading”, “passengers are only allowed one car­ryon and one per­sonal item”, and “the national threat level is orange.”

These announce­ments appear to be on an inter­val loop and never actu­ally rep­re­sent a change in state. Tell me when the threat level goes up, tell me when it goes down, tell me when the white zone is for breakdancing.

Until then, stop with the damn “announcements” and print it on a sign.

2. Mute the TVs

I don’t know about your air­port, but Harts­field, espe­cially the C con­course, always seems to have the TVs turned way the hell up.

Why not offer a couple of options for watch­ing TV:

• Broad­cast the audio on an FM chan­nel, and let those who want to, tune in with a radio.

• Turn on the closed cap­tions so folks can read it if they like

• Sign a spon­sor­ship deal so folks can use air­port WiFi free, or at least free if they want to watch stream­ing video on CNN or Hulu or such.

3. Seats

Again, it may only be a Harts­field thing, but clearly the gate seat­ing capac­ity was scaled for a time when secu­rity lines were shorter and folks could race to their plane right before takeoff.

Air­ports should add more seat­ing beyond the secu­rity check point. I’d envi­sion a large lounge in each con­course, sur­rounded by shops and dining with large status boards.

Even better than status boards, let me sign up for SMS noti­fi­ca­tions for my flight. Alert me when there’s 15 min­utes to go before board­ing, or if my gate changes, etc.

4. Bring your own bin

People doff­ing their shoes, taking their lap­tops out, undo­ing their belts (is this secu­rity or striptease?) cause a bot­tle­neck at secu­rity lines.

Why not give away/sell re-​usable heavy-​duty plas­tic bags that are TSA-​approved — I’m think­ing it’d be sim­i­lar to the Idea bag.

Then folks could stop any place before the secu­rity line, do the secu­rity striptease shuf­fle, put their items in the bag, and walk up to the line ready to go.

After the secu­rity check­point, people would don all their gear again and fold the bag up and be on their way.

5. Sit, then board, then fly

This is my cra­zi­est idea. But hear me out.

Instead of a seat­ing area out­side the gate, imag­ine if you actu­ally sat down in your assigned seat, in your assigned row, and could put your lug­gage into the over­head com­part­ments before your plane arrived at the airport?

Then when your plane arrived, the seat­ing com­part­ment would slide out — allow­ing the arrivals to “deplane”, and the depar­ture seat­ing sec­tion would slide into the plane with every­one and their belong­ings stowed.

It’d reduce turn­around time and garner the air­lines more money — quicker turn, means more flights, which means more fares per day.

Of course it’d require redesign­ing not just all the air­planes, but all the air­ports to make it happen, but hey wouldn’t it be cool!

I was think­ing, in a bad way, about all the TV shows — some good, but most bad — that I watched in my childhood.

And then for some reason unbe­knownst to me, I thought I’d start a blog and inflict this trip to nos­tal­giav­ille on the rest of you.

There’s more about why I started it over at tvofmylife.com

At the time I forked nose itself, but I wasn’t happy with the solu­tion. Folks rightly wouldn’t want to use my fork; I don’t want to commit to keep­ing up with nose’s devel­op­ment; and duh, they have a plugin archi­tec­ture.

So I’m happy to announce nose-​xcover. Install it and it’ll add the –with-​xcoverage option which you can use along­side the built-​in plugin like so:

#nosetests --with-coverage {{ coverage options }} --with-xcoverage
nosetests --with-coverage --cover-package=myapp --cover-tests --with-xcoverage

I updated my Django and Hudson guide to use this instead of the fork.

I hope you find it useful!

In years past it took the form of holiday-​themed appli­ca­tions and dash­board wid­gets.

But this year I’ve got a min­i­mal mac thing going on. So I thought I’d “decorate” with some vin­tage hol­i­day photos.

I ran across Shorpy which has a wealth of vin­tage photos and thought I’d share my selects with you as a gift for the holidays:

• Winter Palace: 1933 — Great com­po­si­tion, and New York, espe­cially, old New York, feels Christmassy.

• Sixth Avenue Shopp­pers: 1903 — Mall schmall, these people shopped with horses!

• Broad­way Santa: 1902 — A reminder that this is a time for giving. Also, Cubanola cigars and hats apparently.

• Wide Christ­mas: 1920 — No matter how much you dec­o­rated your tree, these folks have you beat. Beat.

• Blade Run­ners: 1919 — Great con­trast with the white ice, and the blurry skater in the fore­ground adds a hint of abstractness.

• Kay Jew­lers: 1919 — Jane Sey­mour and her open heart surger^H^H^H^H jew­elry are nowhere to be found.

• Best Christ­mas Ever: 1922 — That kid made out like a bandit, no matter what year it is.

• Five and Ten: 1921 — “Take a look in the five-​and-​ten, glis­ten­ing once again”

• Winter Won­der­land: 1908 — Tell me you don’t want to hop on one of those street­cars with a parcel wrapped up in brown paper. Go on, tell me!

• Iowa City in the Snow: 1940: Pos­si­bly my favorite. It’s straight out of “It’s a Won­der­ful Life.”

I started off using Joe Heck’s great Python and Hudson writeup as a guide.

But to get really good report­ing, includ­ing a cov­er­age report, you’re going to want to use Nose, django-​nose, and my nose-​xcover plugin.

Before you get Hudson up and run­ning, you’ll first want to create a test set­tings module for your appli­ca­tion. Here’s a sample:

With that in place you’ll want to set up a sep­a­rate test require­ments file for pip. If you’re not using pip, I can’t be held respon­si­ble if the cen­tral com­mit­tee gets ahold of you.

Why use my fork of nose? We’ll get to that in a second. First let’s set up our build script.

1. We change into Hudson work­space set up for this build
2. Set up a vir­tualenv for our project
3. Acti­vate it
4. Install our application’s requirements
5. Install our application’s test requirements
6. Assum­ing you have a prop­erly con­structed setup.py, this com­mand will sym­link it into your virtualenv’s site packages
7. Finally run our tests, using our test settings.

The –with-​coverage option tells nose we want to cap­ture cov­er­age infor­ma­tion. The –cover-​package option tells nose we only want report­ing on our appli­ca­tion. The –with-​xunit option will gen­erte a nosetests.xml file in our work­space with the results of the test run.

Finally, the –with-​cover-​xml option is an option I added to nose’s cov­er­age plugin, and the reason why I’ve got my fork listed in our test-requirements.pip. Finally, the –with-​xcoverage option acti­vates my nose-​xcover plugin. It out­puts an XML cov­er­age report that Hudson can use, and it’ll honor the –cover-​package option you spec­i­fied ear­lier, so your cov­er­age per­cent­age won’t be arti­fi­cially low­ered, or inflated, by third-​party code you use.

Now let’s con­fig­ure Hudson to use the two XML reports we’re generating.

First the test pass/fail report.

And then our cov­er­age report, you’ll need the Cober­tura plugin for this:

That’s right, you’ll also want the Chuck Norris plugin. Why? Because Chuck Norris can divide by zero. That’s why.

Also, I’d highly rec­om­mend the Green balls plugin, because Hudson’s default of blue == pass just doesn’t fly with me, or Chuck.

While we’ve been set­ting up Hudson, I had another build in the oven baking using the recipe above. Let’s see how it turned out:

Editor’s note: I updated this post to use my nose-​xcover plugin and not my fork of nose.

For the pur­poses of this I’m assum­ing that you have Python and Django installed. I use Mac­ports but feel free to use Home­brew or any other fine pack­age man­ager.

• Install apache2 and mod_wsgi

~ $ sudo port install apache2 mod_wsgi

• You’ll prob­a­bly want to to add /opt/local/apache2/bin/ to your $PATH in ~/.profile

• Create an httpd.conf file

cd /opt/local/apache2/conf
sudo cp httpd.conf.sample httpd.conf

• Add the mod_wsgi module to httpd.conf

LoadModule wsgi_module modules/mod_wsgi.so

• Make sure the vhosts config file is loaded in httpd.conf

Include conf/extra/httpd-vhosts.conf

• Set up a vhost for your domain in /opt/local/apache2/conf/extra/httpd-vhosts.conf. Note the YOUR-​USERNAME-​HERE place holder.

NameVirtualHost *:80
 
<VirtualHost *:80>
    ServerName local.yourdomain.com
    ErrorLog "/private/var/log/apache2/local.yourdomain.com-error_log"
 
    <Directory /Users/YOUR-USERNAME-HERE/Code/wsgi_apps>
        AllowOverride All
        Options Indexes FollowSymLinks
        Order allow,deny
        Allow from all
    </Directory>
 
    WSGIDaemonProcess local.yourdomain.com processes=1 threads=1 maximum-requests=1
    WSGIProcessGroup local.yourdomain.com
 
    WSGIScriptAlias / "/Users/YOUR-USERNAME-HERE/Code/wsgi_apps/local.yourdomain.com.wsgi"
</VirtualHost>

Note: I amended the WSGI­Dae­mon­Process line above so your code would get refreshed with each new request.

The ~/Code/wsgi_apps path is arbi­trary, that’s just where I keep mine.

Many folks like to keep their .wsgi files in a /public direc­tory inside their Django project. I like to work with­out a project folder, but either way, sub­sti­tute the path to your wsgi file for /Users/YOUR-USERNAME-HERE/Code/wsgi_apps.

• Check your Apache conf before your wreck your apache conf

$ sudo apachectl configtest
Syntax OK

• Set up an alias for your domain in /etc/hosts

  127.0.0.1       localhost
  255.255.255.255 broadcasthost
  ::1             localhost 
  fe80::1%lo0     localhost
  127.0.0.1 local.yourdomain.com #Local Django server
  

• Set up your actual wsgi file

import site
site.addsitedir('/Users/YOUR-USERNAME-HERE/.virtualenvs/YOUR-VIRTUALENV/lib/python2.4/site-packages')
 
import os
os.environ['DJANGO_SETTINGS_MODULE'] = 'mysettingsmodule.local'
 
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()

I’m using Ian Bicking’s vir­tualenv and Doug Hellmann’s kick-​ass vir­tualen­vwrap­per, and you should be too.

If you’re not and you’re able to ignore the cries of all the pup­pies and bun­nies you’re killing because you’re not, then you’ll want to remove the first two lines.

• Fire up Apache

$ sudo apachectl start

That should be it! Go to http://​local.​your​do​main.com in your favorite browser and you should see your Django project.

The instruc­tions for doing all this are on the Inter­net, but I thought it might help others (or at least my future self) to com­pile them all in one spot.

How WordPress caches

There are two layers of caching in Word­Press: * An object level cache, sim­i­lar to Django’s low-​level cache * Page level caching, pro­vided by plu­g­ins, sim­i­lar to Django’s per-​view cache

By the time we’re done, we’ll have both using mem­cache. The object cache defaults to the file system, and most folks use WP Super Cache which caches pages to the filesys­tem as well.

Object caching

• Install the PHP mem­cache module

• Add these set­tings to your wp-config.php

global $memcached_servers;
$memcached_servers = array('default' => array('127.0.0.1:11211'));

You can add as many servers and change the port number to match your mem­cached setup. Example:

global $memcached_servers;
$memcached_servers = array('default' => array('bradfitzrocks:11211', 'bradfitzrocks2.mycompanydomain.com:90210'));

• Install the mem­cache ver­sion of WordPress’s object-cache.php to your blog’s wp-​content folder.

  I don’t want to side­track myself com­par­ing how much Django rocks com­pared to PHP or Word­Press, but I will anyway! In Django, your cache setup is a set­ting, which makes local devel­op­ment soooo much easier.

Stop here and you’ll be live with Word­Press object caching.

Page caching

• You must have fol­lowed the steps above to hook up WordPress’s object cache to memcache.

• If it’s installed, unin­stall WP Super Cache

• Install Bat­cache by putting advanced-cache.php file in your blog’s wp-​content folder, and batcache.php in your blog’s wp-content/plugins folder.

• Acti­vate the Bat­cache plugin from within the admin inter­face, if you’re on Word­PressMU use the “Activate X Site Wide” link.

• Hit reload on a page a few times and then view source and you should see some­thing like this:

<!--
	generated 12 seconds ago
	generated in 0.683 seconds
	served from batcache in 0.006 seconds
	expires in 288 seconds
-->
</head>

• Option­ally, tweak Batcache’s behav­ior by edit­ing the options in advanced-cache.php

var $max_age = 300; // Expire batcache items aged this many seconds (zero to disable batcache)
 
var $remote = 0; // Zero disables sending buffers to remote datacenters (req/sec is never sent)
 
var $times = 2; // Only batcache a page after it is accessed this many times... (two or more)
 
var $seconds = 120; // ...in this many seconds (zero to ignore this and use batcache immediately)
 
var $group = 'batcache'; // Name of memcached group. You can simulate a cache flush by changing this.
 
var $unique = array(); // If you conditionally serve different content, put the variable values here.
 
var $headers = array(); // Add headers here. These will be sent with every response from the cache.
 
var $uncached_headers = array('transfer-encoding'); // These headers will never be cached. Apply strtolower.
 
var $debug = true; // Set false to hide the batcache info <!-- comment -->
 
var $cache_control = true; // Set false to disable Last-Modified and Cache-Control headers
 
var $cancel = false; // Change this to cancel the output buffer. Use batcache_cancel();

The Bat­cache ver­sion I linked to is a GitHub fork of Andy Skelton’s orig­i­nal ver­sion. My fork includes a patch to make sure the Content-​Type header is set cor­rectly on non-​HTML pages. I’m hoping to add some more cache inval­i­da­tion rules on top of the great work Andy already did.

Many thanks to the fol­low­ing folks:

• Mohan­jith for the object cache set up
• Andy Skel­ton for cre­at­ing Bat­cache and being kind enough to drop a ver­sion on GitHub for me to fork
• Casey Bisson for describ­ing the content-​type patch for Batcache

Prob­lem: You’ve got resource or time inten­sive work to be done but you don’t want to do it within the Django request/response cycle. Jobs like fetch­ing remote resources, resiz­ing images, expen­sive data­base queries, etc. are good candidates.

At work, when­ever we have a prob­lem with scal­ing, or really any prob­lem at all, Johnny always asks “What would Brad Fitz­patrick do?”

What he did was build Gear­man. It is a work queu­ing and dis­tri­b­u­tion system. From your client code you send a request off for work to be done, either syn­chro­nously or asyn­chro­nously. Worker instances pull jobs off the stack and per­form them.

The Python bind­ings are stu­pidly simple to use.

# client.py
from gearman import GearmanClient
client = GearmanClient(["127.0.0.1"])
client.dispatch_background_task("echo", "foo")
 
# worker.py
worker = GearmanWorker(["127.0.0.1"])
worker.register_function("echo", lambda job:job.arg)
worker.work()

It’s con­cep­tu­ally sim­i­lar to Rab­bitMQ. Both are mes­sage queues but Gear­man feels a little more spe­cial­ized for use as a task queue, while Rab­bitMQ is more a gen­eral mes­sag­ing framework.

Rab­bitMQ is nice as well but I’ve got one par­tic­u­lar pet peeve with it. When you ask “How do I scale RabbitMQ?” the answer is pretty much “Use the power of Erlang”.

When you ask “How do I scale Gearman?” the answer is “just like mem­cached, add more servers to the clus­ter and spec­ify them in your connection.”

from gearman import GearmanClient
 
client = GearmanClient(["workserver.yourdomain.com",
"procrastinate.yourdomain.com"])

If you’re look­ing for a task queue, I highly rec­om­mend Gear­man — it’s easy to use and easy to scale!