If you keep your home directory under version control, then you may know the sinking feeling that comes from sitting down at work after a late night, looking for that brilliant fix you made right before bed, and finding that your commit is nowhere to be found. What happened?

Upon arriving home, you discover that your final conscious act the previous night was an accidental commit of your home directory — you committed, say, a tiny experimental change to your ~/.profile from last week when you really meant to be committing a patch to ~/big/project to resolve the latest show-stopping bug. You were too tired to notice that the version control system listed the wrong files as you typed the commit message into your editor.

This can, of course, happen if you run git or hg while you are sitting in the wrong directory. But, more subtly, this can happen if you are sitting in the correct directory but call the wrong command. If ~/big/project was, for example, checked out using Mercurial or Subversion, then a git commit in that directory will patiently search up the directory tree, find your ~/.git directory, and look for changed dotfiles or commands to commit instead.

For those of us with several version control systems going at once, in fact, it is a bad idea to have any two projects be “concentric” — where a directory under the control of one version control system wraps another project under different control. I really prefer for each directory in the file system to have exactly one version-control directory in its tree of ancestors.

So how can you safely version control your dotfiles? The solution is to keep your ~/.git or ~/.hg directory out of the way during normal operation — for example, by appending .off to its name — then moving it back into place when you explicitly want to perform a version control operation on your dotfiles. One approach would be to create a shell script like this:

# ~/bin/homedir-git
# Like git, but turns your homedir "on" first

mv ~/.git.off ~/.git
git "$@"
mv ~/.git ~/.git.off

However, I found this approach unwieldy. For one thing, the name of the command was always difficult to type because I am so strongly conditioned to type the plain, unadorned version control command instead. Another problem is that my shell's command-line completion suddenly did not know to offer the usual version-control subcommands because homedir-git or homedir-hg is not a command it recognized.

So my actual solution has been to create a command to toggle home directory version control on and off. That way, I can turn it on; let my normal muscle memory take over as I craft and execute version-control operations; and then turn it off again:

# ~/bin/home-toggle

if [ -d .git.off ] ;then
    mv ~/.git.off ~/.git || exit 1
    echo Home directory version control activated
else
    mv ~/.git ~/.git.off || exit 1
    echo Home directory version control deactivated
fi

Actually, I named my own copy of this shell script ,home because — as you might remember — I name my shell scripts starting with a comma. But you get the idea. And, of course, I check this shell script into version control along with the rest of the suite of customizations that I need on every system where I type.

This whole idea is trivial, obvious, and so simply implemented as to be hardly worth mentioning. But those of us who have been using a Unix environment for decades know that these kinds of tiny micro-customizations for making our lives easier, while they are each so simple, accumulate together into a really amazing result: an environment that offers very little friction because, each time your dotfiles are checked out to your account on a new machine, it instantly becomes an environment where every annoyance — every stone that has ever made you stumble — has already been accounted for and worked around.

The lack of friction at a well-customized command line can be really astounding if you have been putting up with the defaults your entire life. I hope that this simple example encourages you to stop in your tracks the next time something gets in your way for the third, fourth, or hundredth time, and that you will ask: “Could I solve this with a simple shell script?”

It is frustrating that Python's logging module cannot display the tangled tree of configured loggers that often result from combining your own application code with several libraries and frameworks. So I have released a new Python package named logging_tree, which I announced last month during the PyCon 2012 closing lightning talks. My package displays the current logging tree to help debugging, and its output looks something like this:

<--""
   Level WARNING
   Handler Stream <open file '<stderr>', mode 'w' at ...>
   |
   o<--[cherrypy]
       |
       o<--"cherrypy.access"
       |   Level INFO
       |   Handler Stream <open file '<stdout>', mode 'w' ...>
       |
       o<--"cherrypy.error"
           Level INFO
           Handler Stream <open file '<stderr>', mode 'w' ...>

The configuration shown by this tree, it turns out, causes a bug. This diagram helped me fix a real-life application for Atlanta startup Rover Apps, who generously let me open-source logging_tree after I wrote the first version while helping them fix this bug.

In this post I am going to reproduce the problem using a simple 10-line CherryPy application, and then show how I used this logging_tree diagram to craft a solution. But, first, you need to know three things about the Python logging module — and I would like to thank Marius Gedminas for his recent post about logging levels that helped me correct something I had misunderstood.

Read more

I have just released adventure 1.2 on the Python Package Index, an update of my Python 3 port of the original Colossal Cave Adventure game that I announced more than a year ago during the final round of PyCon 2011 lightning talks.

Written in the late 1970s, “Adventure” was the first game to offer players a virtual world to explore at their own pace, driven by their own curiosity. The player directs the game with simple one- and two-word commands like ENTER BUILDING and GET LAMP, together with the cardinal directions — which can, mercifully, be typed as abbreviations (N, NE, E, SE, and so forth, with U and D for up and down). Based on a real-life section of the Flint-Mammoth cave system in Kentucky that the original author helped map, “Adventure” invites you to start collecting treasures from the cave in a quest that eventually involves danger, magic, and even encounters with a few computer-controlled characters, who rustle in the darkness beyond the light of your lamp before finally pouncing.

Keep reading if you want to learn about several discoveries that I made while porting “Adventure” to Python! If you want to know more about the history of the original game itself, I recommend Dennis G. Jerz's admirably thorough paper “Somewhere Nearby is Colossal Cave: Examining Will Crowther's Original ‘Adventure’ in Code and in Kentucky” from the Summer 2007 issue of the Digital Humanities Quarterly.

Playing at the prompt

I was inspired to write the adventure package when I realized that typing a name at the Python prompt could invoke an action if the object it referenced did something useful inside of its __repr__() method.

Read more

]]> http://rhodesmill.org/brandon/2012/one-sentence-per-line/ Tue, 03 Apr 2012 01:25:09 EDT http://rhodesmill.org/brandon/2012/one-sentence-per-line/ One sentence per line, please

I give some advice each year in my annual Sphinx tutorial at PyCon. A grateful student asked where I myself had learned the tip. I have done some archæology and finally have an answer. Let me share what I teach them about “semantic linefeeds,” then I will reveal its source — which turns out to have been written when I was only a few months old!

In the tutorial, I ask students whether or not the Sphinx text files in their project will be read by end-users. If not, then I encourage students to treat the files as private “source code” that they are free to format semantically. Instead of fussing with the lines of each paragraph so that they all end near the right margin, they can add linefeeds anywhere that there is a break between ideas.

The result can be spectacular.

Read more

]]> http://rhodesmill.org/brandon/2012/counting-without-counting/ Fri, 30 Mar 2012 23:34:38 EDT http://rhodesmill.org/brandon/2012/counting-without-counting/ Counting, without counting, in Python

It often irks me that the normal Python pattern for running the body of a loop n times results in the allocation and destruction of n integer objects, even if the body of the loop does not need them.

# Creates one million integers

for i in range(1000000):
    print

# Creates them one at a time

for i in xrange(1000000):
    print

Yes, I know, you will rightly complain that I am too easily irked. The range() pattern is standard. The pattern is simple. The pattern is easy to read. The implementation is really quite fast compared to any real work that I might do in the loop. And, in the bright future when we all use PyPy, the extra million integer objects will be optimized away anyway.

Read more

]]> http://rhodesmill.org/brandon/2012/walking-to-pycon/ Tue, 13 Mar 2012 02:59:10 EDT http://rhodesmill.org/brandon/2012/walking-to-pycon/ Walking to PyCon on your own two feet

Before arranging for a cab ride or airport shuttle, have you considered that your own body is designed for traveling long distances under load?

Last Tuesday I simply walked the 5 miles (8 km) from the San Jose Airport to the PyCon 2012 venue. My route included a stream lined with willow trees, a beautiful nature preserve whose educational displays used wonderful words like “chaparral,” and a glimpse at the games that employees get to play in the back yards of Silicon Valley companies.

What could happen next year, if more people were interested in walking — could we designate a meeting place at the airport, and have a group of PyCon walkers leaving every hour to enjoy the journey together?

5.1-mile nature route from SJC to PyCon

Several people have expressed interest in my walk, so here is a map and details about the route. I will be walking back to the airport on Wednesday morning — probably, judging from the forecast, in my wet-weather gear!

View Walking to PyCon in Santa Clara in a larger map

• Exit the airport, walk across both lanes of Airport Boulevard, and you should find a bicycle trail that runs alongside it. Turn left (north) and start walking along this trail.
• When you get to the sign that forbids you from walking farther, you are ready to turn off of Airport Boulevard. There should be a bridge to your right that crosses the Guadalupe River and then enters a gated parking lot. Cross the bridge and, instead of entering the parking lot, take the ramp to your right that descends the bank toward the river, doubles back, and heads north under the bridge along the east riverbank.

The bridge that turns right off of Airport Drive, crosses the Guadalupe River, and connects to the Guadalupe River Trail. You can see the Trail running right to left beneath the bridge.

• The first stretch of the trail is a bit bleak — there is some green, but also many views of concrete. You will pass under several bridges as cars roar above you, including both bridges for US-101. When you reach the bridge at Trimble Road, you will learn a pattern that continues for the rest of the trail: at each road crossing the bicycle trail splits, with the right fork climbing to meet (and cross) the road, while the left fork passes under the bridge so bicycles can avoid dodging traffic. Unless there is high water, always go under the bridges.
• The long stretch between Trimble Road and the Montague Expressway is more scenic — here, at last, is where I really relaxed and something deep inside of me cried, “California!” Mountains should be visible in the distance to either side. Closer at hand you will see the landscaped campuses of several technology companies. If you are there around lunchtime, as I was, you will see employees playing volleyball, basketball, throwing frisbees, and running on well-manicured trails. It will all look very idyllic. Again, “California!”
• But the real treat comes as you reach the pedestrian bridge where River Oaks Parkway dead-ends into the river. Turn left and cross the Guadalupe on this bridge, turn turn right to continue your hike north past Thamien Park. Soon, you will approach the Ulistac Natural Area. The area around the river becomes even greener, with numerous willow trees swaying in the breeze. (The photo earlier in the article is from this section of the trail.)
• Entering the Ulistac Natural Area was tricky. There was construction, with only a narrow path cutting down to the left between chain-link fences, from the height of the bicycle trail to the level of the river bottoms. But I found my way down, and Ulistac is was really wonderful — there are educational displays about the natural habitat, and several areas where native plants were being grown and protected. If you have time, walk slowly and learn more about the natural landscape in this area of California.
• Wind your way through Ulistac until you come to Lick Mill Boulevard, turn right, and you will finally reach Tasman Drive.
• The last segment of the walk is less scenic. Turn left and walk west along Tasman Drive until you reach the Hyatt and Convention Center. Keep your eyes out for the restaurants on the north side of Tasman; you might be walking back here during PyCon to have dinner! As you climb the overpass where Tasman crosses Lafayette, you will see the Santa Clara golf course ahead to your right and the roller coasters of Great America against the sky to your left — and, of course, yet another great view of the mountains behind and in front of you. The front entrance of the Convention Center on your right will be the first PyCon venue that you reach once you are past the golf course, and the Hyatt and Hilton themselves come next.

Attempt the hike only if you are familiar with walking similar distances at home, if you are sure that you can carry your luggage, and if you bring a water bottle or snack if you will need one. My Rick Steves Convertible Carry On has built-in shoulder straps that convert it into a backpack, which was ideal. But if your bag has wheels you will probably find the trail unmanageable, since most of it is surfaced with gravel. Remember that even if you take more conventional transport to PyCon, you can always take a break later, walk east along Tasman to the Ulistac Natural Area, and enjoy this bit of beauty that Santa Clara offers!

Why do we Python programmers stay so annoyed with JavaScript's broken this keyword? After all, every programming language has rough edges. The problem with this even turns out to be easy to work around once you learn the knack. So why does it feel like JavaScript has committed a fresh offense every time we trip over it?

The answer, I suggest, is that JavaScript manages to disturb very deep mental scaffolds with the behavior of the this keyword. Not all programming language annoyances are created equal. Some involve inconsistency within a language itself, when a pattern set up by one feature is broken by another (think of method names in the Python Standard Library). More serious issues can involve hassles with a language's syntax or poor behaviors within its type system. But in this case JavaScript decided that it would abandon a key property of the system that lies beneath it — that it would break the conventions that, in fact, underlie all programming languages.

JavaScript decided that it would break mathematics.

Let me explain by starting with a simple example. You are already familiar with operator precedence, and how multiplication binds more tightly than addition when both operators appear in the same expression. In the following expression, a will first be multiplied with b, then the result of that operation will be added to c.

There is, in other words, a hidden intermediate result inside of this equation: the result of the multiplication. So (1) is, in fact, a shorthand for writing this sequence of two separate binary operations:

Note that our ability to transform (1) into the pair of lines (2) does not involve any special properties of the operators themselves. This does not illustrate some special feature of multiplication or addition, like the Distributive Property! Instead, we are working down at the lower and more fundamental level of asking what a complex math expression even means. So while we must use argument and proof to learn that addition is commutative, the operators and their precedence are simply a matter of definition — of what we decide it means when we string symbols together to form an expression in the first place.

Now it turns out that the familiar programming language idiom of calling a method in a language like Python or JavaScript is quite precisely analogous to expression (1), because it separates three symbols with a pair of binary operators, where the left operator binds most tightly:

(3)   n = a.b(c)

Until a programmer really grasps what it means for a language to have “first-class functions” — functions that can themselves be manipulated as values — it might be difficult to see that a.b makes quite good sense simply standing by itself. It means “take the a object, look and see whether it has an attribute named b, and resolve the value of that attribute.” And so a.b works perfectly in front of (c) so long as the result of the attribute lookup happens to return a callable.

So expression (3) can be decomposed like expression (1), and in Python the following two steps are exactly equivalent to statement (3) — except, of course, for defining an extra local variable fn:

(4)    fn = a.b
       n = fn(c)

This is, again, simply a property of how expressions work in math — of the fact that you ought to be able to compute intermediate results by pulling an expression apart into its constituent binary operations. But, alas, JavaScript decided to break this property of expressions, and makes extra invisible magic happen when the two operators are used in combination — magic that does not happen when they are separated into separate steps. Or, perhaps there is a more interesting way to think about it:

/* In JavaScript this is NOT a pair of binary operations. It is a SINGLE
   ternary operator that, to sow confusion among programmers, happens to
   use the same symbols as two well-known binary operations. */

a.b(c);

/* This ternary operator is roughly equivalent to: */

var fn = a.b;
var old_this = this;
this = a;
fn(c);
this = old_this;

Younger programmers, for whom a.b(c) is simply a gesture, may find our distaste for JavaScript's behavior inexplicable. The problem is worst for the experienced programmer or mathematician, who — every time she types it — remembers what the dot and parentheses really mean as clean and separate operations, but has to remember that their meanings change when they appear in combination. This semantic instability flaunts a very long tradition of defining math operators so that expressions can be composed together and broken down again without changing their meaning.

And that, I think, is why it annoys us: because from early grade school through college we have learned that math expressions compose and decompose cleanly, and JavaScript takes that symmetry away.

One last note for newer Python programmers reading this: you might be suspecting that Python itself has some kind of magic involved here, because how else could it remember later whether you had pulled method fn off of the specific object a instead of off some other instance of that class? The answer is that every lookup of an instance method returns a new object, called a bound method, that remembers the object on which the lookup took place.

>>> class C:
...     def __init__(self, n):
...         self.n = n
...     def __repr__(self):
...         return 'C%d' % self.n
...     def fn(self, m):
...         return self.n + m
...
>>> a = C(100)
>>> b = C(220)
>>> a.fn
<bound method C.fn of C100>
>>> b.fn
<bound method C.fn of C220>
>>> b.fn(5)
225

What about your own least favorite language features, whether in JavaScript, Python, or something else? Are they all simply about scruples and inconvenience? Or can you identify some deep-seated assumptions of your own mental scaffolding that keep ruining your experience with a specific language? Let us know in the comments!

By this time tomorrow I will doubtless be wearing swimming trunks — in northern Ohio — in winter — while listening to people rave about Java and C# and Ruby and wondering what I have gotten myself into. I will be at CodeMash, a conference that was started by developers who wanted an event that focused solely on the programmer while including a wide range of languages and technologies.

It was lucky that I signed up the moment that CodeMash 2012 registration opened back in October, because the 1,200 conference tickets sold out in only 20 minutes — no event in the Python community had prepared me for that kind of demand!

I learned about CodeMash conference from its co-founder Brian Prince when we were fellow speakers at PyOhio last July. So why did a Python programmer like me decide to attend?

• Even though the technologies that Brian chooses are different from mine, he is clearly animated by the same passion for combining good code with good community. If his co-founders are at all like him, then I knew that a great event was in the works.
• I want to learn another conference's culture. For example, I was stunned to see strong suggestions on the CodeMash mailing list that attendees should not carry laptops — instead, they recommend pencil, paper, and something they call Sketchnotes. Since good teachers have a reflex that makes them try to bring the whole audience along with them as they make a point, we could actually be slowing up PyCon speakers when half the audience is face-down typing and clearly a half step behind what is being said.

• I could predict that moving to Bluffton, Ohio, as winter descended would leave me with very few opportunities to meet other developers. After several weeks of being the only programmer I know, a large regional conference will let me bask in the company of other people who understand what I do for a living.
• It is too easy to judge other languages by what I think are drawbacks in their design, or by the poor code that most programmers produce whatever their language. I want see what excites the real experts who solve interesting problems using Java, Ruby, and C# — people who use those languages to the hilt, and do a great job of it.
• Being evangelized by smart people is interesting and humbling, if you let down your guard and really listen. And hearing Ruby and C# people explain the glories of their languages will remind me of how I must sound when I hold forth on the advantages of Python, or Emacs, or Vibram Fivefingers.

• Brian wants to make CodeMash more popular for Python programmers — and a few luminaries like Bruce Eckel, Mike Pirnat, and Mark Ramm are already on the schedule this year. Next year I might offer to speak. But first I wanted to show up and just listen, figure out the vibe of the conference, and learn more about what is happening outside of the Python community.
• Finally, it does sound like great fun: eating, drinking, and being merry at an indoor water park resort in the middle of an Ohio winter. Kalahari, here I come!

(Images are Creative Commons licensed from Flickr photographers iriskh and Alan.Barber.)

(My official Concentric CSS “style.css” is in a GitHub repository)

Perhaps my mind is unusually visual, but I have no idea how people clearly picture the effects of their CSS rules when following the common advice to sort their properties alphabetically within each declaration block (as in the answers to this Stack Overflow question).

The main alternative to alphabetization seems to be a CSS property order that claims to be based on the box model but which tackles properties in a sadly haphazard order. It starts with display and position, which are a great start, but these are immediately followed by the height and width — even though these dimensions only apply to the content, which is nestled down inside the deepest level of the box! The order then continues to skip around wildly, jumping outside the box to talk about the margins, then jumping almost all the way back inside to talk about the padding. Finally it gets to the borders, even though they will actually be sandwiched between the margins and the padding when rendered by the browser.

For reference, here is the box model as illustrated in the CSS standard itself:

The fact that width and height apply only to the content, and not to the padding or anything else, is a crucial issue in CSS — an issue that causes newcomers a lot of grief. For everyone's sanity, I need the order in which I declare properties to reflect the structure of the box itself, with the width and height coming after the decorations that wrap around them.

I call the result Concentric CSS and the basic template looks something like this:

{
    display: ;    /* Where and how the box is placed */
    position: ;
    float: ;
    clear: ;

    visibility: ; /* Can the box be seen? */
    opacity: ;
    z-index: ;

    margin: ;     /* Layers of the box model */
    outline: ;
    border: ;
    background: ;
    padding: ;

    width: ;      /* Content dimensions and scrollbars */
    height: ;
    overflow: ;

    color: ;      /* Text */
    text: ;
    font: ;
}

Are you surprised that I sandwiched background between the border and padding instead of saving it until later? My reasoning is that the padding is the first part of the box that actually gets painted with the background color — I have positioned background so that everything beneath it gets the background color, while everything above has its own color (borders) or is transparent (margins).

I am not certain about my choice of where to place the overflow property. Its current position is dictated by the fact that it is often triggered by too small a height and width — so it often serves as a kind of “else clause” to say what should happen if the height and width are too constraining, and so the property makes sense next to them. But one of the consequences of overflow can be that scrollbars appear around the box, and an argument could be made for putting overflow up between the borders and padding, because that is exactly the place where the standard insists that scrollbars be drawn.

For the sake of completeness I have written up my preferred property order — with a much more exhaustive list of properties than in the sketch shown above — as a style.css file that lives in its own GitHub repository. I welcome comments and improvements, especially ones that will make this ordering an even better reflection of the box model. I want my CSS to make as much sense as possible both to the newcomer as well as the seasoned professional!

The Compass CSS authoring framework has become one of the standard tools that gets installed when I start working on a new web application. I always version-control not only the .scss Sass source files that I myself write, but also the .css CSS files that Compass compiles from them. That way, anyone who checks out the project immediately gets a working web site without having to install Compass — or even having to know that it exists, if they are not themselves involved in writing the CSS.

But those of us who work on the CSS do need Compass, so I have a standard technique that I copy from project to project that installs Compass into a small, local Ruby environment, providing the same kind of isolation and reproducibility that Python has taught me to know and love thanks to virtualenv.

The process goes like this.

First I create a compass/ directory, typically up at the top level of the project, that initially contains nothing but a pair of shell scripts. The first script, install.sh, knows how to get Compass downloaded and installed:

#!/bin/bash
# compass/install.sh - install Compass under the "./Gem" directory

if ! which gem >/dev/null ;then
    echo 'Error: no "gem" command available'
    echo 'Please "sudo aptitude install rubygems1.8" or "ruby1.9.1"'
    exit 1
fi
BASE=$(dirname $(readlink -f $(which "$0")))
cd $BASE  # the directory where this script lives
gem install -i Gem compass
gem install -i Gem compass-susy-plugin

When this shell script is run, a compass/Gem/ directory gets created with the compass command down inside of it. To let me forget where the binary is located and how it can be safely invoked, a second shell script named compass.sh wraps up the details:

#!/bin/bash
# compass/compass.sh - properly invoke the "Compass" program

BASE=$(dirname $(readlink -f $(which "$0")))
export GEM_HOME=$BASE/Gem
export RUBYLIB=$BASE/Gem/lib
$BASE/Gem/bin/compass "$@"

I always tell my version-control system to ignore the compass/Gem/ directory and the 5,115 files that Ruby creates beneath it; instead, I simply remember to re-run install.sh after each fresh checkout when I want to get to work on the CSS.

When I first start a project, I need an initial set of .scss files to start editing, which can be supplied through the Compass command create. Usually I want the CSS output to be written somewhere outside of the compass/ directory, so my initialization steps go something like this:

$ cd compass/
$ ./install.sh
$ ./compass.sh create
$ rm -r stylesheets/    # where Compass wanted the output CSS!
$ emacs config.rb
# There, I adjust the line: css_dir = "../webapp/static/css"
# Plus, I also tend to set: line_comments = false
# Of course, use your editor-of-choice for this step!

I check into version control the config.rb file and the compass/src/ directory of pristine .scss files. I tell my version-control system to ignore the .sass-cache/ directory that gets created inside of compass/ each time the CSS is rebuilt.

Once those changes checked in, I never need to run the create command again for the whole lifetime of the project. Instead, whenever I am ready to work on the CSS, I simply open the .scss files in my editor and in another window I do this:

$ cd compass/
$ ./compass.sh watch

Just as a good development web server will auto-restart when you finish editing a file and hit Save, the Compass watch command will sit patiently all day and re-compile your Sass files to CSS whenever it sees you change them. By leaving it running, I can edit a .scss file and then hit Reload in my browser almost immediately to see the result without having to stop and run any intermediate commands.

The files, therefore, that wind up in my version control are these:

compass/compass.sh
compass/config.rb
compass/install.sh
compass/src/_base.scss
compass/src/_defaults.scss
compass/src/ie.scss
compass/src/print.scss
compass/src/screen.scss

And, of course, I also version-control the CSS files that Compass writes as output, over where I put them in the “main directory” of my web application:

webapp/static/css/ie.css
webapp/static/css/print.css
webapp/static/css/screen.css

This all works out extremely well. Normal developers and deployers look under the webapp/ directory and see what looks like normal, if preternaturally well-written and organized, CSS files exactly where they expect to find them. Only we CSS geeks on the project know that they are written by shuffling over to the Compass directory and firing up a program that compiles them from simpler source files. And thanks to my handy little pair of shell scripts, the Compass install itself is always completely automated.

Of course, as you can tell from the error message that install.sh prints if it cannot find Ruby, these scripts are aimed at an audience using Ubuntu or Debian. Please note in your comments to this post how you might expand those instructions to help people using Fedora, MacOS X, or other operating systems!

Today I got stuck between a rock and hard place — or, more specifically, stuck between the assumptions of Robert Brewer and those of Ian Bicking. In case you ever try mounting a WSGI application underneath a larger CherryPy application, here is the story.

# Easy: putting a WSGI `app` behind the CherryPy HTTP server

server = CherryPyWSGIServer(('0.0.0.0', 8001), app, numthreads=30)
server.start()

# More interesting: mounting `app` beneath a particular URL path
# This works, but `app` gets no logging or error handling

cherrypy.tree.graft(app, '/api')
cherrypy.tree.mount(None, '/static', {'/' : {
    'tools.staticdir.dir': static_root,
    'tools.staticdir.on': True,
    }})
cherrypy.engine.start()
cherrypy.engine.block()

$ pip install flup
Downloading/unpacking flup...
$ python -c 'import flup.middleware'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named middleware

$ pip install wsgilog
Downloading/unpacking wsgilog...
ImportError: No module named ez_setup

# Transform bare `app` into one that logs and 500s on exceptions

from paste.exceptions.errormiddleware import ErrorMiddleware
from paste.translogger import TransLogger

app = ErrorMiddleware(app, debug=debug_flag)
app = TransLogger(app, setup_console_handler=debug_flag)

# Now we proceed as before to build our CherryPy application.

cherrypy.tree.graft(app, '/api')
...

# from paste/exceptions/errormiddleware.py

class ErrorMiddleware(object):
    ...
    def exception_handler(self, exc_info, environ):
        ...
        return handle_exception(
            exc_info, environ['wsgi.errors'],
            ...)

# from cherrypy/wsgiserver/__init__.py

class WSGIGateway_10(WSGIGateway):

    def get_environ(self):
        """Return a new environ dict targeting the given wsgi.version"""
        ...
        env = {
            ...
            'wsgi.errors': sys.stderr,
            ...
            }
         ...
         return env

# Adding three middlewares: error, logging, and my own

app = ErrorMiddleware(app, debug=debug_flag)
app = TransLogger(app, setup_console_handler=debug_flag)

errlog = open('http-tracebacks.log', 'a', 0)

def app2(environ, start_response):
    environ['wsgi.errors'] = errlog
    return app(environ, start_response)

cherrypy.tree.graft(app2, '/api')

What was my big idea? That in printed Python code, indentation should be visible.

Any of you who have had to read many Python listings printed in books will immediately recognize the problem that I wanted solved. When a listing is long enough to run on to a second page, it is often less than clear whether the code there is continuing at the same level of indentation, or whether the page break has just happened to correspond to a point in the code where it dedented out to the previous level. Languages with braces or explicit “end” statements to close blocks never have to worry about this. But in Python — especially where a script or code snippet ends without ever returning to the outermost level of indentation — the last few lines of the script feel as though they are left hanging if they stand alone at the top of a new page of text.

Of course, there were several practical considerations that had to be settled. A symbol for indentation had to be chosen, for example. I selected the Unicode double-chevron because it is a character that is never valid in actual Python code. Then the publisher had to be convinced to try the experiment; it helped that I had the full support of my editor, Laurin Becker, who also prepared the layout people for the fact that these chevrons were not part of the code and would need to remain visually distinct from it. Finally — because I had no desire to insert and color each chevron by hand — I had to write an lxml script to insert the chevrons into my OpenOffice documents, then go back and ruefully remove by hand the chevrons that got nonsensically inserted into snippets of other languages like HTML.

But now I want to hear what readers think! I have not yet seen a review that mentions whether the visible indentation helps, hurts, or is simply irrelevant to the Python listings. If you happen to have seen the new edition of Foundations of Python Network Programming, let me know what you think. While creating the effect cost some time and effort, I will happily do it again in my next book if turns out to have actually helped readers scan the program listings.

Over the years I had already built a stable full of shell functions for doing grep(1) and find(1) in various combinations, and my functions even defaulted to using pcregrep(1) when available because Perl-style REs (which have nearly the same format used by Python) seem to require so many fewer backslashes for common cases. It hardly seemed worth the effort to move to another tool — and the grin output format seemed glarish and offensive at first glance, not at all like the austere output of traditional grep:

$ pip install grin
$ cd Python-3.2b2
$ grin autoGIL
./Doc/whatsnew/2.6.rst:
 3167 :   :mod:`autoGIL`,
./Misc/HISTORY:
 5577 : - There's a new module called "autoGIL", which

But I forced myself to try using it again a few weeks later, and when actually doing real work I could not help but notice that the results were far easier for my eyes to scan than grep output. This is an important point: so often, the issue of whether something looks like it will be easy to read is a quite different matter than whether it is actually difficult to read the moment you stop looking at it and start trying to look through it to see your data! Now I can clearly see where the output from one file ends and the next begins — which traditionally is quite difficult if you are looking through a series of files with very similar names. Long file paths no longer push matching lines to the right, splitting them across the right edge of my standard 80-column terminal window. In fact, one of my first objections to its layout had been the line consumed by each stand-alone file name; but in practice, I found, far more lines are gained because of the matches that are not split across the right edge.

You can run grin against individual files or directories by naming them on the command line; adjust the set of files to which it pays attention; and even revert its output to a traditional file:line format if you want to pass the output to another program. You can, in other words, ask it to behave more like traditional grep. But its default behavior, I gradually realized after running it several hundred times, is by far the most common way that I had been using grep anyway — to search the source files beneath my working directory for a pattern.

The author, Robert Kern, has answered emails promptly and was happy to accept a few patches. I now have a shell script that I run whenever I set up a new Unix account that builds a virtual environment, includes its bin directory in my PATH, and installs grin as one of the most important Python tools that I need always at my fingertips. Try it out!

My long labor is at last at an end: Apress has published my rewrite of John Goerzen's Foundations of Python Network Programming (2004) and I have released the example programs as both Python 2 and Python 3 code in a Bitbucket repository and also as a zipfile on the Apress web site. I can finally return to more casual writing — this is my first blog post in many months! — and can be more active in Python projects again.

I was amazed, as I studied the book's popular first edition, at how far Python has come since 2004. None of the modern Python web frameworks existed; the book offered one chapter on CGI programming, another on mod_python, and never even mentioned Zope. Python 2.3 was the most recent version of Python. The book could cite FTP as one of the “most widely used protocols on the Internet.” Parsing HTML involved stepping through the markup with HTMLParser and the recommended approach to processing XML was the awkward xml.dom. JSON was not yet popular enough to even warrant mention. The author recommended the dependable syslog module over the newfangled logging package in the Standard Library, and remote administration tools like paramiko and fabric either did not exist or were not yet on the radar.

Looking back, it appears that 2005 — the year following the first edition's publication — was a turning point for Python web technology in particular. CherryPy saw its first release in late 2004, and over the next year we were introduced to Django, Pylons, and Turbogears, all on the heels of the Ruby on Rails “framework” phenomenon. And both of the web scraping technologies covered in my new edition of the book, BeautifulSoup and lxml, also came out in 2005. In the face of all these changes, the web chapters received a full rewrite based on the latest technology.

I also rewrote the introductory chapters so that UDP and TCP each get their own chapter, focused on the protocol itself, instead of leaving the information about each protocol split between a client socket chapter and a server socket chapter. I have replaced the separate chapters on forking, threading, and event-driven servers with a single chapter that takes an example network server, rewrites that same server six different ways — including Twisted and a raw polling event loop — and then uses funkload to study the resulting performance. And even though an editor objected that it lacked unity, I insisted that a chapter be dedicated to memcached, message queues, and map-reduce, since they are so important to scaling modern Internet services.

Apress had already slated the project for Python 2 based on the advice of a well-known Python programmer, and I heartily supported his recommendation. His wisdom is bourne out by the result: as you can see from the README.txt that I included with the source code bundle, fully one-quarter (22/86) of the programs will simply not run today under Python 3 because of missing dependencies. I see this new Foundations of Python Network Programming as the perfect companion for Python 2.7 — which was released as I was writing — since the book will bring you up to date with the final features of the Python 2 series and the third party libraries that support it.

I enjoyed working with the Apress team; Laurin Becker is a great managing editor who was always on my side, and Michael Bernstein as technical editor gave great feedback which made this an even better book. There were only two annoyances that I encountered with the book's release. First, as the accompanying image illustrates, Apress used a much more compact layout than in the first edition. This not only makes the book appear much shorter than its predecessor, but also makes it (to my eyes) visually crowded and more tiring to read. Second, Apress was not prompt about changing the project title after their decision to target Python 2, with the result that Amazon had “Python 3” in the title all the way through the book's release in late December — resulting in 1-star reviews for the first few weeks!

I will, by the way, soon be blogging about my experience rewriting the book's example code to work under Python 3. This should help two groups of people: anyone porting a Python 2 network application to Python 3 will probably run into the same issues I did; and people who are already experimenting with Python 3 will want to know how to apply the book's techniques to the newer version of the language.

Feel free to submit questions about the book or its example programs to the Bitbucket source code repository where the code is stored. The revision took far more effort than I had expected, but I am very happy to have found a format into which I could distill so much of my own — and the Python community's — network programming experience, so that it will help programmers for years to come. Enjoy!

IOError: [Errno 24] Too many open files

An lsof against the process confirmed that more files were open after every restart. My guess was that leftover references were keeping a few Python file objects alive from one CherryPy reload to the next, and I remembered having once used a neat utility for exploring the object heap. After some investigation I re-discovered that it was Heapy. So I added a pdb breakpoint, did some investigating, and was puzzled to find that after each restart only four Python file objects existed — one for each log file in our application.

(I also tried using objgraph, which is far easier to use than Heapy, but it could not tell that there were file objects in memory at all. I have no idea why.)

Well, this was a puzzle. How could the number of open file descriptors increase without bound when Python was clearly deleting all of the old file objects? The answer, once I finally tried reading the source code to the Autoreloader plug-in, was of course very simple: CherryPy performs each restart by doing an exec() to completely wipe out the process image and replace it with a new instance of the CherryPy application. Which is certainly a very thorough approach! Except for one thing: file descriptors in Unix are set, by default, to survive an exec() call, but the new instance of Python that spins up inside of the process does not know that they are there, so they never get closed. It appeared that suddenly calling ØMQ out of existence with an exec() also left a few sockets lying around.

Several possible solutions came to mind. What if a more thorough effort was made to delete all Python objects before running the exec() call? That sounded daunting, though — it might take a lot of effort to march through all of the application object trees closing everything down. I could have focused my efforts just on finding the file objects, but that approach felt fragile; what would happen the next time one of our developers wrote a new module that opened a log file?

Monkey patching the open() built-in to create files with their close-on-exec flag already set is, of course, too terrible a solution even to contemplate.

In the end, the simplest solution seemed to be the creation of a little CherryPy plug-in that, as the very last shutdown action, would loop over all existing file descriptors and set their close-on-exec flag. Here is the plug-in, in case the pattern helps anyone else:

"""Make sure file descriptors close when CherryPy exec's."""

import os
import sys
from cherrypy.process.plugins import SimplePlugin

class CloexecPlugin(SimplePlugin):
    def stop(self):
        if hasattr(sys, 'getwindowsversion'):
            return
        import fcntl  # not available under Windows
        max = os.sysconf('SC_OPEN_MAX') if hasattr(os, 'sysconf') else 1024
        for fd in range(3, max):  # skip stdin/out/err
            try:
                flags = fcntl.fcntl(fd, fcntl.F_GETFD)
            except IOError:
                continue
            fcntl.fcntl(fd, fcntl.F_SETFD, flags | fcntl.FD_CLOEXEC)

    stop.priority = 99

Of course, I suspect that this problem was happening all along, even before we added extra logging and then integrated ØMQ into our application. But back then, with maybe only one or two stray file descriptors surviving each restart, it would have taken five hundred or a thousand CherryPy restarts for the problem to be noticed — and, apparently, none of us developers ever reached that impressive total. Now we know to be careful!

Several images disappeared from the Our Fresh World green-building web site because Google changed their Picasa API recently — and I must not be subscribed to the proper mailing list or blog to have been warned ahead of time. Where are incompatible Google API tweaks announced?

The site owner and his photographer use Picasa web albums to upload, edit, and maintain their image collection. They simply give special tags to their favorite photos, and my application code then knows that it is supposed to display those photos on the web site. I almost used Flickr for this application, both because I am an avid Flickr user myself and because I consider its web interface more usable. But, perhaps predictably, Picasa had the much stronger search API — whereas you can either ask Flickr for the photos in a particular set, or ask for all of someone's photos that share a particular tag, Picasa lets you can combine the two queries and ask for only the photos that are in a particular set and that also share a specific tag. And since search is what attaches pictures to this web site, Picasa was my choice.

All was going well, with each page getting populated by searches like:

http://picasaweb.google.com/data/feed/api/user/name
?kind=photo&tag=solar-power

Then I received an email from the site owner, complaining that many of the photographs had disappeared! After seeing some complaints in the Picasa forums about recent versions of the user interface treating certain “special characters” in tags as spaces instead, I suddenly wondered whether the hyphen in several of our tags (like the “solar-power” tag in the URL above) was the cause of our trouble. I adjusted my code so that this search became:

http://picasaweb.google.com/…?kind=photo&tag=solar+power

And, voilà, the images returned and were again visible! Does anyone know what forum or blog I should have been following to be informed of this critical change by Google? It is dismaying to have a site break in front of a customer when the very reason that I chose a Google product was because of their powerful API for integrating my application.

But then, eventually, one has to write the actual review.

And so, a full four months after that friendly email from Packt Publishing, it is time that I sit down and put together some thoughts about Carlos de la Guardia's first book, Grok 1.0 Web Development. Carlos is a long-time veteran of the Zope and Plone communities, and Grok, of course, is the web framework that places a simple and agile convention-driven engine atop the otherwise notoriously XML-ridden Zope application framework. Grok is an important project, because it packages the technology of Python's oldest and most experienced community of web developers in a way that makes it easy to extend and use.

Read more

I have worked out an elegant solution by combining two of the powerful user-space filesystems available through the FUSE mechanism in the Linux kernel. Let me take you through the story of how I put them together!

Read more

I have wanted to try the multiprocessing module out for some time, and now have a consulting project that will really benefit from multiple processes: they will let our application run third-party plugins without having to worry that any bugs or indiscretions which they commit might damage or hang our main server, which can remain safe in another process.

First, one can only stand in awe at the achievement — and the amount of work — that the multiprocessing module represents. I cannot imagine the time that it would have taken our team to figure out all of the differences between Linux and Windows when it comes to processes, shared memory, and concurrency mechanisms. In fact, the approach we are taking might not even have been feasible under those circumstances. By figuring out how to get locks, queues, and shared data structures all working cleanly on such different architectures, the multiprocessing authors save Python programmers out on the street like me from reinventing a dozen wheels when we need to support multi-platform concurrency.

Well, almost.

There is one rather startling difference which the multiprocessing module does not hide: the fact that while every Windows process must spin up independently of the parent process that created it, Linux supports the fork(2) system call that creates a child processes already in possession of exactly the same resources as its parent: every data structure, open file, and database connection that existed in the parent process is still sitting there, open and ready to use, in the child. Consider this small program:

from multiprocessing import Process
f = None

def child():
    print f

if __name__ == '__main__':
    f = open('mp.py', 'r')                                                      
    p = Process(target=child)
    p.start()
    p.join()

On Linux, the open file f keeps its value in the child process; the child has inherited an open connection from its parent:

$ python mp.py
<open file 'mp.py', mode 'r' at 0xb7734ac8>

Under Windows, however, where the multiprocessing module has to spawn a fresh copy of the Python interpreter to which it gives special instructions to just run the function f(), the module is a clean slate without an open file inside:

C:\Users\brandon\dev>python mp.py
None

Now, my complaint is not exactly that the multiprocessing documentation is misleading on this point; under its section on Programming guidelines, it makes it quite clear that:

On Unix a child process can make use of a shared resource created in a parent process using a global resource. However, it is better to pass the object as an argument to the constructor for the child process.

I have no quarrel with this advice; if I am careful to pass everything the child needs in its list of arguments, then I can be sure that my code will work under both Linux and Windows.

But I do wish that the multiprocessing module provided more support for testing this condition more rigorously under Linux. In particular, I wish that there were some way of turning the simple forking logic off — of saying, “Yes, I know that Linux will let you create a child process very simply using fork(2), but for my sanity would you please create the child process from scratch like you do under Windows so that I can test whether my code accidentally depends on residual state from the parent process that I did not see that I was using?” I looked at the multiprocessing "forking.py" module to see whether I could turn on the Windows-style process spawning even from inside of Linux, but the mechanism is chosen by a bare module-level check of "sys.platform" and if I overwrite that variable with 'win32' the code then dies when it tries to import "msvcrt" which is available only under Windows.

There is, thus, even in principle, no way that I can test my multiprocessing application under Linux which will give me any assurance that my child processes are not accidentally taking advantage of data structures and open connections left lying around by the parent process; only by actually moving over to Windows itself can I see how my child code really behaves on its own. I have created a feature request in the Python bug tracker to see whether this situation can be improved.

But even with this one inconvenience — which is troubling me much less, now that I at least understand why my application was behaving so differently under Windows — the multiprocessing module is still a huge leap forwards for Python programmers who need to run code in heavyweight processes with all of the isolation and safety that they provide. Thanks again to Jesse and the multiprocessing team!

How can a collaborative site like ours best be edited and updated? Well, I would like to report some modest initial success with an experimental approach: I now maintain the site as a Sphinx-powered documentation system stored in a BitBucket repository into which I pull changes made by my collaborators. The advantages are several.

• The change management tools supported by traditional CMS systems, even at their best, seem somehow anemic when compared to the toolkit provided by a good DVCS like Mercurial. Where, for example, does even a capable CMS like Plone provide anything like Mercurial's “backout” or “blame” commands?
• Markup as well-designed as reStructuredText is not only a lot of fun to use, bit it also very cleanly separates content from design. Authors working in plain text tend to produce clean, readable content without the messy markup often associated with visual HTML editors, or, worse yet, the disaster that is Microsoft Word.
• Staging — a feature I find essential, but which seems missing from many default CMS configurations — occurs automatically! Each author can see locally how the site will look with their changes, and after doing a pull I can review the site's appearance on my laptop before finally deploying the new content to the production site.

To top it all off, authors get to use their own editor-of-choice when making contributions, and we all get extra practice cloning and merging in my favorite DVCS. I am optimistic about this direction, but I will post again if we wind up hitting snags in the future. Finally, of course, feel free to clone our repository if you want to see how Sphinx looks when running a generic web site.

>>> import sys
>>> len(sys.modules)
35
>>> foo
...
NameError: name 'foo' is not defined
>>> len(sys.modules)
225

That's 190 extra modules — merely importing them takes around 60 ms on my laptop! Where are they all coming from? And how could an exception cause so many imports, including such illustrious modules as email, mimetools, and xml?

After reading Ubuntu's sitecustomize.py file and all of its consequences, the situation became clear. Their apport crash-reporting subsystem instruments Python with an exception hook that, when invoked, discovers that my system says enabled=0 in my /etc/default/apport file and so it undertakes no special crash logging. But, on the way to loading the routine that performs this simple check, it performs two quite flagrant and unnecessary imports, pulling in both apt (that brings with it 83 packages) and apport (an additional 107 packages).

The solution? I have removed the python-apport package, along with the ubuntuone-client suite that depends on it. After the uninstall, exceptions are — wonderfully enough — not causing a single import of a new module! Now, finally, I can continue writing my import hook in peace.

firefox -remote 'openURL(http://example.com/, new-tab)'

But after a few months of manually cutting and pasting URLs into Chrome — which wasn't actually that bad, since the address bar in Chrome is such a convenient and large target — I decided that I needed a real solution. After not finding anything like a -remote option, I discovered that Chrome can at least be run with a debugging port open:

google-chrome --remote-shell-port=9222

The protocol that Chrome speaks is primitive enough that it was quick work to implement a small client in Python. Rather than merely cutting and pasting its code here on my blog, or even be satisfied with making it available on bitbucket, I decided to place the code inside of a new Python package and make it generally available on PyPI as chrome_remote_shell.

Thanks to this simple package, a four-line program (not counting the shebang and comment) is now all that I need to ask Google Chrome to open a new tab:

#!/usr/bin/env python
# Name this file "google-chrome-open-url"
import sys
import chrome_remote_shell
shell = chrome_remote_shell.open()
shell.open_url(sys.argv[-1])

To teach Emacs to start using Google Chrome when I clicked on a link, I only needed to supply it with two new settings:

(setq browse-url-browser-function
      'browse-url-generic)
(setq browse-url-generic-program
      "google-chrome-open-url")  

And now everything works. I hope that these notes prove useful to someone else. Enjoy!

It was with regret that I tendered my resignation yesterday as the Editor-in-Chief of Python Magazine. While the publisher will keep producing the magazine by distributing PDFs on the web site, the transition to the new format has dragged on long enough — both for both myself and our customers — that I have run out of enthusiasm. My last responsibility will be to shepherd the February and March issues through the publishing process and safely on to the PDF readers of our subscribers.

I hope that the authors featured in the October issue will forgive me for not writing my usual blog post last year touting their achievements; I had just received the sad news that the publisher could no longer afford the rising costs of printing and shipping Python Magazine, and I did not want to further advertise the magazine until its fate was certain one way or the other.

I have by no means been a perfect editor. In particular, the publisher hoped that I would get the magazine — which was running eight weeks late — back on schedule. Instead, my bumbling first month as editor made the magazine an additional week late, and by the time I hit my stride in May it was another week behind. Although the schedule then stabilized at a steady ten weeks late, I never did manage to start reeling the fish back in. The only metric, I suppose, which I can really claim to my credit is that I oversaw a nineteen-fold increase in the number of em-dashes in the magazine — 247 appeared over the course of 2009, up from only 13 the year before!

I should express thanks to my co-workers: Arbi, Emanuela, and Cathleen are smart, helpful, and professional, and were patient with me as I learned the ropes. Doug Hellmann gave me ample training as he handed over the reins, and also supported the magazine later as an acquisitions editor. Several associate editors performed solid reviews of incoming articles. And, of course, the greatest privilege of being Editor-in-Chief was to help such a wide array of voices from the Python community find their way into print — from Steve Holden, the illustrious chair of the Python Software Foundation, to young Meran Cambpell-Hood, an eleven-year-old from New Zealand who described using Python for the first time to process data for her science fair project.

Which reminds me: the authors from the October issue never got their moment in the spotlight! The article by Meran Campbell-Hood about her science fair project was the most fun to edit, but every single article was interesting and taught me something. Steve Holden interviewed James Tauber about the secrets of a successful Python start-up; Yusdi Santoso finished his two-part series on the Python program he wrote to produce the PDF for the beautiful EuroPython brochure last year; the original editor of Python Magazine, Brian Jones, returned to talk about why he now tends to choose Django for web projects rather than PHP; and Joe Amenta introduced his "3to2" project, which will help Python programmers support their old Python 2 users while still moving ahead with the transition to Python 3. Finally, Greg Newman explained how to turn Emacs into a powerful Python IDE, and Steve Holden and myself rounded out the issue with our usual editorializing.

With many of the rest of you, I am eager to see the debut of the new Python Magazine web site. And I look forward to seeing everyone at PyCon 2010 in less than three weeks! While I will not have the joy that I did last year of walking the halls of PyCon as a newly-minted Editor-in-Chief, able to make dreams come true and grant the fame and fortune of being a published author, I will at least enjoy being a developer among developers in the best programming language community on Earth!

The September issue of Python Magazine appeared on the web late last week and only now, as a new week has started, am I finally sitting down to announce it! The articles range from technically heavy development topics to high-level thoughts about the whole Python community, with plenty in between.

I have to say that our prettiest article this month is “Using Python to Create Beautiful Documents” by Yusdi Santoso, who shares the basic secrets to document generation that he learned when building the EuroPython 2009 brochure using a Python program. Traditional typesetting and computer typography were both interests of mine when I was growing up, so it was fun to read Yusdi's introduction to using ReportLab to generate PDF documents. I look forward to his follow-up article that we will soon be publishing, on the specific techniques that he used in creating the EuroPython booklet.

The other technical articles are an introduction to using SOAP in Python; a guide to displaying objects in a Mac OS X GUI created with PyObjC; an article introducing Python's own built-in Tkinter GUI toolkit; and a small excursion of my own that attempts to explain the popular “trick” (well, it really confused me the first time I saw it!) of defining a decorator using a pair of nested functions. I should confess that my own article contains what is probably this issue's biggest mistake, as pointed out quite promptly by alert reader Emanuel Woiski: in the code sample that is its whole crux of my example, I somehow managed to omit one of the most crucial lines, shown here in bold:

def log(function):
    def log_wrapper(*args):
        print "called %s%s" % (
            function.__name__, tuple(args)
            )
        <b>return function(*args)</b>
    return log_wrapper

I suppose I will now need remedial cut-and-paste training of some sort.

Finally, the issue is rounded out by three articles that move back from Python coding and step out to wider vantage points. Justin Lilly provides an excellent guide to customizing your Vim setup so that it becomes a powerful Python integrated development environment. Steve Holden muses about why diveristy is so difficult and reveals some of the recent goings-on surrounding the diversity statement that the Python Software Foundation has been working on. And my own editorial seeks to point any Python Magazine readers who do not yet have a strong connection with the wider community in the direction of greater engagement with the world of Python.

All in all, I think the issue is a nice mix of fact, experience, and opinion. Please consider subscribing if you would like to hear more about what people are doing with Python, and how. I enjoy reading it; so might you.

I wanted to measure distances in Tolkien's Middle-earth. While a flat map distorts such measurements, it occured to me that Google Earth can correctly measure both lines and paths across the curved surface of the globe. I soon found excellent documentation for using image overlays with Google Earth, so I downloaded a map of Middle-earth and tried placing it on the globe.

Imagine my disappointment when I saw the result shown in the above image! At first I made the mistake of not holding down the Shift key when resizing the image in Google Earth; the Shift key is absolutely critical for the image to maintain its aspect ratio as you stretch it to the right dimensions. But even after learning this habit, it was painfully clear that the Middle-earth map's projection was different from that expected by Google Earth: the map is far too narrow at the top.

Obviously, it was time to pull out Python, my favorite programming language, and see whether the Python Imaging Library could help me make short work of converting a map from one projection to another.

Read more

My Python table at the Atlanta Linux Fest. You can also watch a short video of me demonstrating a depth-first search to some students who dropped by the table. (Thanks, Richard Davies, for the video!)

Running the Python table at the Atlanta Linux Fest this past weekend was a really incredible experience.

First, there was the great feeling that the pillars of the Python community were standing behind me as I stepped forward to represent my favorite programming language. It was Andrew Kuchling who noticed that exhibitor tables at the Fest were free for non-profits like the Python Software Foundation, and Steve Holden who forwarded me a heads-up since I live in Atlanta (the Fest had not yet made it on to my radar). The inimitable Aahz personally shipped me the promotional kit, including a huge “Python” banner and stacks of brochures, that he himself had just used at OSCON 2009. And, completing the loop, it was Andrew who followed up to ask if there were any last things that I needed, and sent me a pile of over one hundred Python stickers that wound up being very popular at the Fest. (I returned home with exactly one, which is sitting next to me on my desk as I type this!)

Here are some lessons that I learned from the experience:

Read more

I would like some advice from Zope and Plone folks about how to create forms that are not only easy for other developers to specialize, but which allow several specializations to be composed together. While I have used zope.formlib and z3c.form before for simple tasks, I have not yet been able to tell whether they support these more advanced kinds of operations.

Some background: I am doing some work on GetPaid for Plone with the generous funding of Derek Richardson who, if his dreams had not carried him away from grad school at the end of the Spring semester, would have tackled this same work as part of the 2009 Google Summer of Code.

The current mechanisms that GetPaid provides for customizing its checkout process are very primitive, and my task is to improve them. That is why I have been thinking about customizing forms.

Read more

The August issue of Python Magazine is out, and the cover issue is one of the most exciting that I have had the privilege to publish. Following up on his popular talk at PyCon 2009, professor of computer science Dr. Bill Punch has written an article for us with his colleague Dr. Richard Enbody about replacing C++ with Python in Michigan State University's introductory programming class.

The difficulty they faced was that the second and subsequent classes in Michigan State's curriculum have continued using the C++ language; only the initial course could change to Python. While non-majors taking the introductory course get to learn Python and leave, the computer science majors have to then start the second course without any specific knowledge of C++. How much, skeptics wondered, would it hurt the Python students to be a full semester behind in their C++ knowledge compared to students who took the “old” C++ version of the introductory course?

The circumstances at Michigan State wound up being perfect for making the change to Python a full-fledged, statistically valid experiment to determine how much student grades would be hurt because of the switch. Read the article for the detailed results, but I can report — with great satisfaction — that the Python students proved themselves the equals of their C++ peers, and also that very exciting and unexpected results ensued among the students whom the study was not looking at: the non-majors, for which this was their only programming course, and the seasoned grad students, who rolled up their sleeves and learned Python in order to be able to TA the course. Again, see the article for all of the details!

What else do we have this month? Articles about how easy concurrency becomes with Stackless Python; about documenting your software projects using the powerful Sphinx documentation system; about creating simple graphs with the industry-strength matplotlib plotting library; a guide to the details of hashing algorithms, and how they relate to the Python dictionary implementation; and, finally, the usual thoughts and ponderings from both myself and from Steve Holden of the Python Software Foundation. Come join us!

The problem, of course, was the chance of collision. Because my shell script names tended to be short and pithy collections of lowercase characters, just like the default system commands, there was no telling when Linux would add a new command that would happen to have the same name as one of mine. This was actually not very likely on, say, a System V Revision 3 workstation in the 1980s, but the trouble became quite a bit more acute when I moved into the world of Debian. Red Hat never really worried me, because they packaged (comparatively) so little software. But Debian today supports a huge number of commands; my modest Ubuntu laptop shows several thousand available:

$ apt-file search -x '^/usr/bin/[^/]*$' | wc -l
21733

The solution was obviously to adjust my command names in such a way that they were still easy to type, but would never be chosen as system command names. For me, “easy to type” means not having to use the shift key, and very few characters turned out to be available, unshifted, on a modern keyboard. The lower-case letters are the very characters used in system commands; brackets, backslashes, the colon, the back-tick, and the single-tick all had a special meaning to the shell; and the slash and dot characters both mean something special in a filename. (The slash divides directory names from filenames, and thus cannot appear in a filename itself, while the dot means “hide this file from normal browsing” if it leads the name, and separates a file from its extension in many other cases.)

There was but one character left: the simple, modest comma.

A quick experiment revealed in a flash that the comma was exactly the character that I had been looking for! Every tool and shell that lay in arm's reach treated the comma as a perfectly normal and unobjectionable character in a filename. By simply prefixing each of my custom commands with a comma, they became completely distinct from system commands and thus free from any chance of a collision.

And, best of all, thanks to the magic of tab-completion, it became very easy to browse my entire collection of commands. When trying to remember which of my commands are available in my ~/bin/ directory on a given system, or when simply trying to remember what some of my commands are called, I simply type a comma followed by tab and my list of commands appears:

$ ,«tab»
,complete-scp        ,go-thpgp            ,range
,complete-ssh        ,gr                  ,svn-store-password
,coreoff             ,hss                 ,umount
,coreon              ,mount-thpgp
,find                ,mount-twt

I heartily recommend this technique to anyone with their own ~/bin/ directory who wants their command names kept clean, tidy, and completely orthogonal to any commands that the future might bring to your system. The approach has worked for me for something like a decade, so you should find it immensely robust. And, finally, it's just plain fun.

I am home from a relaxing vacation to the Midwest, and while I was gone last week my excellent publishing team released the July issue Python Magazine to the world. I am particularly pleased that two of the feature articles in this issue come from important segments of the Python world that we have not heard much from in previous issues.

First, IronPython, the .NET version of Python for Windows, is the topic of Jonathan Hartley's article about acceptance testing. He illustrates that, regardless of the language in which you write your .NET application, you can deploy simple strategies to make your application testable through a Python test harness, and thereby bring Python's strong flexibility as a testing language to bear on a product that you might be writing in another .NET language.

Second, Malthe Borch, a veteran of the Zope community, shares how he has written Chameleon, one of the fastest template language implementations available for Python. By processing each template and turning it into Python bytecode before it is ever used to render a single page, Malthe eliminates a huge amount of redundant processing as that same code is used over and over again. His library is a key ingredient in the new high-efficiency web frameworks appearing in the Zope world. His work might even (fingers crossed) become one of the components that the Plone community uses as they streamline their framework and move towards a lighter and more agile “Plone 4”.

Other technical topics covered are: the Hadoop map-reduce framework; the concept of hash functions, and how they apply to Python; and the new string formatting operator which Guido hopes will replace all of the percent-signs that currently litter our code. Wrap it all up with an editorial by Steve Holden about EuroPython 2009 and an editorial by me, and you have a complete issue! If you do not find Python Magazine sitting on your local newsstand, then I hope you will avail yourself of a subscription and, as always, let me know what topics you would like to see covered in future issues. Enjoy!

Thanks to more than an hour of work today, I have a pretty list of a few dozen commands that make it easy for a WebFaction account holder to install the powerful lxml Python package for parsing HTML and XML under their hosting account. You can read Ian Bicking's wonderful blog post “lxml: an underappreciated web scraping library” for more information on why you want to be using lxml instead of any of its alternatives.

So, why do I say “drat”?

First, because I just tried out my instructions on another of my WebFaction accounts, and there the extra steps weren't even necessary; this other server of theirs already had lxml's dependencies installed! I suppose, had I been a bit more patient, that this support ticket that I glanced over this morning would have inspired me to ask WebFaction to install the libraries lxml needs on the server where I myself was working. But it felt like some sort of offense against symmetry to rely on something that WebFaction doesn't install everywhere, and I was perhaps just in too big of a hurry. Which, of course, cost more time in the end.

The other reason I say “drat” is because, now that I look at Ian's post again after all these months, I see that he has instructions for making the package install its own dratted copies of the system libraries it needs! Too bad that lxml's own installation instructions omit this crucial piece of information.

How typical, and how predictable. It turns out that I just needed to listen to Ian Bicking more carefully. How often we fail to do that, as individuals and as a Python community. Listen to Ian Bicking, everyone. Listen.

Read more

WebFaction has even finally wrested domain name control from my fingers. Gone are the installations of bind that used to be running everywhere, and the smattering of shell scripts with which I wrapped “nsupdate” to publish new host names. Now, I accomplish host name changes in a few moments through the control panel on WebFaction. This gives me more time to do creative things for my customers, and gives me one less daemon that I have to keep upgraded if I want to avoid security problems. (If you want to try them out for yourself, using this WebFaction sign-up link will credit me as the one who recommended them.)

This move to making WebFaction my default choice for hosting new client projects is only the most recent chapter in the very long story of my devolving responsibilities on to other people. The previous steps went something like this:

Read more

Further hypothesis — this relationship is non-linear. My guess is that frameworks which take, say, four seconds to restart probably get no more IRC attention than frameworks which take two seconds to restart, so adding mere seconds of delay to your framework's startup routine is not going to generate thriving community. There is, instead, a threshold — probably around ten or twelve seconds — above which the delay feels so long that the developer can, in good conscience towards their employer, switch over into IRC during a restart without feeling like they hurt their productivity.

The NTLK project wants to support installing their package as a Python egg, so they asked me to tidy up their setup.py file and prepare everything for distribution via the Python Package Index.

As usual, my desire for simple and reproducible behavior when distributing Python packages has run aground on the tangled magic for which setuptools is so justly famous.

Read more

Before detailing its content, I should apologize that the magazine keeps arriving later and later each month. Alert readers will have noticed that this “June” issue is actually appearing on July 9th. It is late because, in these inexperienced first few months as Editor-in-Chief, I have been treating magazine editing as a serial process, and have not really been starting work on an issue until the previous one is out the door. To fix things, I will start working more like an efficient, modern, pipelined microprocessor, and fetch articles for one issue while a second issue is being edited and yet a third is in layout to be published.

So, watch for a gradually improving schedule over the last half of this year! One reader wrote me to ask whether the late schedule was a sign that the publisher was losing interest in the publication. In fact, it is the person on staff who is most dedicated to Python — myself! — who is at fault, so no lack of interest in the publication is at all implied.

So, what does the “June” issue have to offer?

Plenty!

Read more

Obviously, Python would be e. e. cummings — the poet for whom whitespace was most truly significant!

What can you look forward to reading? This issue worked out as a balance between community news and several sophisticated technical articles on using Python better. The technical articles include:

Read more

Now, think about the Python programming language, and try the same shift in perspective. Yes, we are all happy that readers of Linux Journal have selected Python as their Favorite Programming Language in this year's “Reader's Choice Awards” and we are pleased when we move up a few more percentage points in something like the TIOBE Programming Community Index. But comparing our market share with that of other programming languages can, too often, fool us into thinking that we are playing a zero-sum game in which our own community can expand only by prying other programmers' fingers off of the languages they currently know and love.

Try something else. Imagine all of the world's people, and ask: what percentage of them use Python? And what might it look like to increase that percentage? You will suddenly think of the grade school down the street, where you could volunteer one afternoon a week teaching Python programming at their computer club. You will realize that the local community college provides a simple IT curriculum, but gives their students no information about getting connected to a local programmer user's group. You will begin to wonder what difference simple Python skills might make in the hands of teenagers, college students, and young families raising children.

You will, in other words, start thinking about how to change the world.

And the reason that I'm excited about the April issue of Python Magazine is that our cover stories focus on Python programmers who, during the recent high-profile United States elections, made this kind of step outward into realizing what Python could do in their communities.

• Mitch Trachtenberg has written our feature article, about how his ballot scanning software helped election volunteers and officials in Humboldt County, California, discover that nearly 200 ballots had not been counted by their Premier Elections Solutions voting systems. Thanks to his efforts, California has now de-certified that particular software from being used in elections in their state.
• We also have an interview with Neal McBurnett, whose ElectionAudits software automated the statistical calculations in Boulder, Colorado, that allowed their election audits to focus on close races where more ballots have to be checked before a winner can be certified with confidence.
• To celebrate the fact that April 22nd was World Plone Day, we feature an Introduction to Plone with instructions for quickly downloading, installing, and customizing this popular Content Management System.
• Our more technical articles describe how to create your own domain-specific language with a parser that gets invoked when you run import on a file with an extension of your own choosing, and how to create a protocol simulation framework that lets you experiment with how packets will traverse a network. And Mark Mruss offers another Welcome to Python article that this time covers Python dictionaries, whose usefulness and elegant design were certainly a big part of my own adoption of the language.
• Finally, Steve Holden shares his perspective as head of the Python Software Foundation, this time reflecting on the wonderful success of PyCon 2009 last month.

I hope you'll take a look at this month's issue, and I hope that you learn as much from reading it as I myself learned from the process of editing. Enjoy!

After years of writing verbose and repetitive setup.py files for my Python packages, I am unable to write another. Instead, I have started writing Pyron, a tool that gathers the same information by inspecting a Python package itself. Not only does this mean that I get to stop repeating myself, but that my projects will become much more uniform because package metadata will be represented through common conventions instead of explicit (and repetitive) configuration. Though Pyron is still very primitive, it has already allowed me to reduce simple packages to only a README.txt plus their actual Python source code.

Read more

Watermarked page (click for PDF). The light blue design is a PDF file that pdftk resizes and centers on the target document.

My last challenge was that, on certain pages, the watermark we were using had to be turned upside down. “Simple,” I thought, “I'll use pdftk to turn the watermark over before applying it!” I just had to process the watermark image with the letter S (“south”), which tells pdftk to rotate the image by 180°, and then use the result as the watermark:

$ pdftk arecibo2.pdf cat 1S output arecibo3.pdf
$ pdftk in.pdf background arecibo3.pdf output wmark3.pdf

Upside-down watermark (click for PDF). Whoops! After turning the watermark upside down, pdftk lost the ability to properly center it.

Read more

After two months of being tutored in the arts of magazine publishing by retiring editor Doug Hellmann, March has been my first month in the Editor-in-Chief chair of Python Magazine. It is exciting to have my first issue come out while I am here at PyCon 2009 in Chicago. I am talking with other programmers, meeting new friends, and, of course, eyeing everyone I meet with the question of whether they might be suitable author or technical editor material.

I am especially excited about our cover issue this month! “Commanding Robots with LEGO Mindstorms” shows how simple it is for a Python program to manipulate both a binary on-the-wire protocol and binary calls into a Windows DLL, all without ever having to leave the Python standard library! For me, this is the real magic of Python: that it not only introduced an incomparably clean syntax and tidy language feature set, but that developers of both the standard library and of third-party Python modules are committed to creating vastly simple interfaces for what in other languages can be very difficult problems. The article is a great guide to using best practices and powerful tools when linking Python to other protocols and libraries.

Other articles include “Getting Started with Message Queues” which talks about how to arrange your application around a central queue so that you can distribute expensive work across dozens of machines; “Statically Analyzing Python Code” by the author of PySmell about how Python's built-in code parsing tools can be used to start investigating powerful ideas like type inferencing; and “Using Python for Pedigree Analysis” that is yet another success story from the world of science, about how Python — which began to be adopted very early in its history by working scientists — continues to move into new areas wherever science needs a clean and powerful language to replace the tangle of low-level code and temporary scripts that traditionally characterize systems written by those whose first expertise is not software design.

Throw in our several regular columns and my own monthly editorial, and you have a complete issue. (I think it was while writing the editorial that it really sank in that I am now an Editor-in-Chief!) I hope you will consider subscribing, and I especially hope you will subscribe to the print edition — for only an extra dollar a month, you will receive an actual, physical artifact that you can leave in the breakroom at work, share with a friend who wants to explore Python, or leave at a client's site to expose their own employees to the world of Python and its community. And, say hello to me here at PyCon!

Then I digressed with a blog entry on a slightly different topic, nested list comprehensions in Python, because I happened to write one while creating the image we will use as our sample watermark. It shows the famous Arecibo space message, and is a tiny image of only 23×73 pixels that looks like this when enlarged:

The basic watermarking process itself is very simple thanks to a wonderful tool that I discovered called pdftk (short for “PDF toolkit”) which, as usual, Debian has already packaged for me. It can rotate documents, extract pages, concatenate several files together, and help fill out PDF forms from data in a file. Of particular interest here is its ability to either “stamp” an image on top of each page of a document, or to place one in the background as a watermark.

The watermark image itself has to be a PDF file — pdftk does not deal in any other file formats — which is why I needed GraphicsMagick to convert the Arecibo image into a PDF in the first place. Putting the two steps together, one has a primitive but workable process for using a PNG image as a watermark:

$ gm convert arecibo.png arecibo.pdf
$ pdftk in.pdf background arecibo.pdf output wmark1.pdf

Hefty watermark (click for PDF). A first attempt at watermarking results in a huge watermark that reaches both to the top and bottom edges of the page.

As you can see, pdftk automatically adjusts the size of the watermark image to reach precisely to the edges of the page being marked — which is a huge favor given the difficulty I would have had in resizing the watermark myself to match the page size of the input file. But, in the above case, the result seems less than perfectly attractive; watermarks usually sit tidily near the center of a page, rather than running all the way against its edges.

Clearly, we want to add some margins to the watermark. And though margins are easy to add to some image formats — they would be simple to add to the arecibo.png file that we are using in this example — in actual practice I need to support watermarks that might be in vector formats like SVG or EPS. While I could go through each possible input format and contrive some way of adjusting its margins, it would obviously be much more convenient to convert everything to PDF first, and then add margins directly to the PDFs.

I used Debian's apt-cache search command to look for additional tools that might help me (which is how I found pdftk in the first place!) and found an old command called pdfcrop that was part of the texlive series of packages; it supports a --margins option with which whitespace can be added around a PDF file. But I found that it often would refuse to process a perfectly good PDF file with a horribly uninformative error message like:

Error: Cannot move `tmp-pdfcrop-10631.pdf' to `out.pdf'!

I tried to investigate the error message, but discovered that pdfcrop is actually a Perl script that writes LaTeX macros which are then run against the target PDF file. And it was last updated in 2004. I have, alas, elected not to make it part of my toolchain.

Then I discovered that Python itself has a quite serviceable PDF package named pyPdf, with the bonus that it is written in pure Python and therefore requires no external libraries! Thanks to its ability to adjust the “bounding box” that defines the edges of an image in PDF coordinates, adding margins was as simple as loading the image, doing some addition and subtraction, and then saving the result. To add modest 10-point margins to the Arecibo message, for example, we can create this wmargins.py script:

from pyPdf import PdfFileWriter, PdfFileReader
pdf = PdfFileReader(file('arecibo.pdf', 'rb'))
p = pdf.getPage(0)
for box in (p.mediaBox, p.cropBox, p.bleedBox,
                                    p.trimBox, p.artBox):
    box.lowerLeft = (box.getLowerLeft_x() - 10,
                     box.getLowerLeft_y() - 10)
    box.upperRight = (box.getUpperRight_x() + 10,
                      box.getUpperRight_y() + 10)
output = PdfFileWriter()
output.addPage(p)
output.write(open('arecibo2.pdf', 'wb'))

You can test this yourself by installing pyPdf in a convenient temporary directory with virtualenv, running the above script, then calling pdftk on the result:

$ virtualenv vpython
$ vpython/bin/easy_install pyPdf
$ vpython/bin/python wmargins.py
$ pdftk in.pdf background arecibo2.pdf output wmark2.pdf

Watermark with margins (click for PDF). Margins prevent the watermark from reaching the page edges, which allows the blocks of text to assume the role of defining the visual shape of the page.

All pretty simple, right? Well, it turns out that there was one final complication — and that, before I was finished, I actually wound up spending more than an hour reading the PDF specification in order to understand what, exactly, was going wrong! But that will be the topic for my last blog post in this series. Stay tuned.

The form [... for x... for y...] nests, with the last index varying fastest, just like nested for loops.

Have you ever seen a Python list comprehension like that, with two or more for loops inside? I have just written my first one! It was only recently that I discovered they were even possible, when I encountered several in a draft of the upcoming Natural Language Processing with Python book. (Which should be great — watch for O'Reilly to publish it!) They almost never turn up in other code that I encounter, and perhaps for good reason: they were deeply confusing the first time I saw them!

The code I have just written is shown below. It uses the Python Imaging Library to produce an image I will use in the series of blog posts that I started yesterday on watermarking PDF files. The code requires a small arecibo.txt file, detailing the radio message that was sent from the Arecibo Observatory in November 1974 to any other civilizations that might be listening. As you can see, I have successfully used two for clauses in the list comprehension that generates the image's pixels:

"""Draw the Arecibo message (blue on transparent)."""
from PIL import Image
image = Image.new("RGBA", (23, 73))
image.putdata([
    (192,224,255,255) if char == '1' else (0,0,0,0)
    for line in open('arecibo.txt')
    for char in line.strip()
    ])
image.save('arecibo.png')

Each pixel is a four-value tuple, by the way, because an RGBA image not only has a red, green, and blue channel for each pixel, but also an “A channel” specifying its opacity or transparency. The colors in use here are a completely opaque light blue, and a completely transparent color (the four zeros). The result looks something like:

My mistake in reading the multiple for clauses was that, old C-language programmer that I am, I was expecting the comprehension structure to be concentric. That is, I thought that the last for must “enclose” the ones above it, creating a mess of lists inside of lists inside of lists. But it turns out that they are much simpler to read than that. Just read them like normal loops, with the “big loop” described first and the subsequent loops nested inside of it:

# The list comprehension said:
  [ <i>expression</i>
    for line in open('arecibo.txt')
    for char in line.strip() ]

# It therefore meant:
for line in open('arecibo.txt'):
    for char in line.strip():
        <i>list</i>.append(<i>expression</i>)

So, to read the comprehension, just picture colons appended to each for clause and, finally, the expression moved down inside of the innermost for loop.

Now that I have made this conceptual leap, I can “picture” the normal for loops each time I see a complicated list comprehension, and they are trivial to read and write! It still, I admit, feels odd that the expression, which would be deep inside of normal for loops, goes in front of them in a comprehension instead. And I am not sure that double comprehensions should become part of my normal coding style. (How many other Python programmers understand them? Has everyone else been using them without problems?) But they are a neat trick to have up my sleeve when I need to iterate over an image quickly and want to pack everything into a single, easily-bloggable expression.

But I will begin the series rather simply, with the first two lessons that I learned during the project:

1. Always verify PDF correctness with Adobe Acrobat.

The trusty Xpdf viewer, with which I have viewed PDF files for years, turns out to have a remarkable ability to decipher and display even somewhat damaged PDF files. That's a great feature — if someone else has produced the PDF, and you just need to read it, whether it's damaged or not. But this “feature” becomes a problem if you have just produced a PDF, and want to know about any errors in the file before your customer does!

In this situation, Adobe's Acrobat Reader should be your viewer of choice. Not only is it probably the software that your customers will be using anyway, but it is — and this seems intentional on Adobe's part — a very stringent interpreter. The error it displays for a corrupt PDF, I must admit, is among the least-informative I have seen this month:

There was a problem reading this document (14)

But the information that this does yield is invaluable: your customer will not be able to see or print this PDF until you find the bug in your toolchain and fix the result. It is, of course, objectionable and problematic to have to install a closed-source product on what might otherwise be a completely clean development system; but I have found it irreplaceable for its ability to show me problems before my customers see them.

This mistake cost me more time than you might imagine. The tool chain I built for my customer generates several intermediate PDF files, and it turns out that the error was happening fairly early in the process — so that one of the first files produced, and therefore each subsequent PDF, was invalid and could not be opened with Acrobat. But because I was viewing them all with Xpdf, I spent many minutes looking at the final step in the chain and wondering why that PDF tool was often dying, when the PDFs it was consuming looked so good in my viewer!

2. Avoid ImageMagick 6.3.7 when producing PDFs; try GraphicsMagick instead!

More recent versions of ImageMagick might not have any problem producing PDF files from PNG images, but the ImageMagick that currently ships with Debian unstable, version 6.3.7, seems to have routine problems trying to turn some of my customer's PNG files into valid PDFs. To avoid having to compile, install, and maintain my own version of ImageMagick, I cast around for an alternative, and was startled when Google brought me to the GraphicsMagick project! Here was ImageMagick done right: instead of creating dozens of commands on your system, as though this were the 1970s, GraphicsMagick defines a single gm binary with multiple sub-commands:

$ gm convert watermark.png watermark.pdf

Check out their web site for more great features; but I'm simply happy that the PDFs it has produced so far have been clean, correct, and consistent.

A question for my readers: can a good, open-source PDF checker be found somewhere, that is at least as stringent as Adobe Acrobat? Leave a comment below if you have a suggestion; such a tool would have made this project considerably easier!

$ sudo easy_install package_url

Read more

1. Run the following script in your home directory. (You might want to use less to read the output.)
2. Ignore files whose date does not reflect your own activity.
3. List the oldest files in a blog post and discuss!

#!/usr/bin/env python
"""Print last-modified times of files beneath '.', oldest first."""
import os, os.path, time
paths = ( os.path.join(b,f) for (b,ds,fs) in os.walk('.') for f in fs )
for mtime, path in sorted( (os.lstat(p).st_mtime, p) for p in paths ):
    print time.strftime("%Y-%m-%d", time.localtime(mtime)), path

Only include files whose last-modified time is a date on which you really touched the file. The file's time should neither result from an error (a few files beneath my own home directory have an incorrect date of 1970-01-01), nor from unpacking someone else's archive that has old files inside of it. For example, I myself have excluded the following pair of nearly 17-year-old files because their dates reflect their age inside of the Python 3.0 source archive, instead of the actual moment last month when they became part of my home directory:

1992-03-02 ./src/Python-3.0/Demo/scripts/wh.py
1992-03-02 ./src/Python-3.0/Tools/scripts/dutree.doc

But there is no requirement that the actual content of each file you list be your own. Whether you wrote the file yourself long ago, or downloaded it from some ancient and forgotten FTP site, you have a story to share!

Within the rules given above, here are the oldest files beneath my own home directory:

Read more

My mistake was to exaggerate somewhat the verbosity with which PyFlakes reports a syntax error. My email to Chris, in fact, made the following rather extravagant claim:

The problem is that … on a syntax error, PyFlakes prints out an error message, then the entire contents of the module that it cannot import, and finally a line that contains a number of spaces equal to the offset into the file of the syntax error …

Just glancing at the PyFlakes source code is enough to see that this accusation cannot be true:

    try:
        ...
    except (SyntaxError, IndentationError):
        value = sys.exc_info()[1]
        (lineno, offset, line) = value[1][1:]
        ...
        print >> sys.stderr, 'could not compile %r:%d:' \
            % (filename, lineno)
        print >> sys.stderr, line
        print >> sys.stderr, " " * (offset-2), "^"

Clearly, this simply prints the line that the Python exception cites as having caused the problem, followed by a primitive attempt to position a ^ character at the location of the error. For simple syntax errors, this actually produces output which is identical to that of normal Python:

$ python error1.py
  File "error1.py", line 3
    return x y
             ^
SyntaxError: invalid syntax
$ pyflakes error1.py
could not compile 'error1.py':3:
    return x y
             ^

But the PyFlakes behavior is quite different from that of the standard interpreter if the syntax error happens in a Python statement that has been continued across several lines of source code. Imagine that there are two functions in a file, and that we have started typing a docstring for the first one but have not yet closed the triple-quote:

def square(x):
    """Returns the square of x.
    return x * x

def cube(x):
    """Returns the cube of x."""
    return x * x * x

Here, Python and PyFlakes give quite different reports:

$ python error2.py
  File "error2.py", line 6
    """Returns the cube of x."""
             ^
SyntaxError: invalid syntax
$ ~/.emacs.d/usr/bin/pyflakes error2.py
could not compile 'error2.py':6:
    """Returns the square of x.
    return x * x

def cube(x):
    """Returns the cube of x."""
 ...76 spaces... ^

Do you see what has happened? The unterminated triple-quoted string looks as though it ends several lines later, at what we ultimately intend to be the beginning of the next triple-quoted string. (If the file instead contained no further triple-quoted strings, then the new string would appear to extend all the way to the end of the file.)

This is no problem for the Python interpreter itself, which modestly displays only the final line of the multi-line syntax error. But PyFlakes, not checking for this possibility, prints out the entire triple-quoted string, followed by a ^ character that is indented the entire length of the triple-quoted string. In the example that I was testing last night, this produced a line of nearly four thousand spaces that then ended in the lone little caret character.

So while PyFlakes is certainly more verbose than standard Python, it is not being nearly as profligate as I claimed. It does not insist on printing out your whole module, but limits its output to lines that actually appear involved in the error. Either way, it is the following line — the one that starts with all of the spaces — that is really the problem, since too many spaces send one of the Emacs Flymake regular expressions spiralling into exponential oblivion.

Perhaps Flymake should support a command-line option that omits code snippets entirely, so that Emacs will have only error messages to process. Either way, Chris and I can be more productive now that we can safely integrate a patched version of this excellent tool into our coding sessions.

While the first twenty revisions of my branch deal with simple Python 3.0 mechanics, I must congratulate Mr. Schüttel for most of the improvements in the subsequent dozen revisions. We not only exchanged emails all day as I produced one revision after another that I needed him to test, but he then joined me on IRC tonight until 3:30am in his time zone as we worked out the last problems.

The issues were all related to localization under OS X. The astronomy library underlying PyEphem used the C-language functions sscanf() and atof() to turn strings into numbers, and it turns out that these functions are very sensitive to locale under the specific combination of OS X and Python 3.0! Because of his locale, the functions wanted a comma instead of a period to separate whole numbers from their decimal fraction (so that π would have to be input like 3,141 rather than like 3.141). They also wanted month name abbreviations to be in German rather than English, ruining my test cases that check planet positions against Naval Observatory tables which use English month abbreviations like Jan, Feb, and Mar with very little regard for how the months would be spelled in German. We are still mystified by the combination of Python version and operation system that were necessary to cause this problem:

Python 2.6 under Linux: tests passed
Python 3.0 under Linux: tests passed
Python 2.6 under OS X: tests passed
Python 3.0 under OS X: broken: sscanf()/atof() change with locale

But the tests worked fine if he put LANG=C on the command line.

To work around this problem, I discovered a wonderful PyOS_ascii_strtod() function in Python's C library that avoided all of the problems that I was having with localization, and so I gradually rewrote the astronomy routines to use that function instead. Fixing the problems with month names was easier; instead of trying to make Python convert month abbreviations by passing the '%b' conversion character to time.strptime(), I simply converted months to integers myself and then passed the integers in with the simpler '%m' format character. It was only late at night that we finally tracked down every routine that was misbehaving.

The final puzzle was how to release my new software. The Python Package Index does not yet allow a project to offer separate source code archives for both the Python 2.x and the 3.0 version of a project. There might be some clever way of storing both versions of the source code in the same .tar.gz file, but the packages I see on PyPI that already support both 2.x and 3.0 contain enough #ifdef statements to convince me that I want to keep my branches separate!

I was saved by a peculiarity of my project. Though it provides the Python package named ephem, the actual project on PyPI has always been named “PyEphem” instead. The product has to have the “Py” in front, you see, to distinguish it both from the original text-screen ephem command and the modern XEphem graphical application, and my first instinct had been to name the project after the product. But as I gained experience with PyPI, it seemed more and more awkward that programmers who wanted their programs to be able to:

import ephem

had to remember to type something different when installing:

easy_install pyephem

Since I had been wanting to switch PyPI names anyway, I suddenly saw my chance! I have now released the new Python 3.0 version of PyEphem under the actual package name ephem and will continue to maintain the 2.x version of the package separately under the old pyephem name. Obviously, this solution only works because of my project's unique circumstances; I have no idea what other projects should do who also want to come out with their new Python 3.0 versions this weekend.

Fans of PyEphem should rest assured that when I develop new features, I will be adding them to both versions of PyEphem for probably at least the next decade. I have absolutely no intention of abandoning or slowing development of the Python 2.x version of the library; I simply wanted the library available to users of the new platform. As always, feel free to email me with bugs, suggestions, and new features — and, enjoy using PyEphem and Python!

The answer is that, while the necessary changes were surprisingly easy, they took lots of time to figure out because I did not find them documented in any one place. I offer the following notes to assist any other adventurers who want to experiment with porting their extension modules to 3.0. These notes might also suggest useful additions to the official documentation.

But, first, I need to issue three cautions. To develop under 3.0, you may have to forego several Python tools that you probably thought you could no longer do without. The world of 3.0 is a windswept and icy landscape from which the glaciers have just receded, and you will find the stone tools rather primitive when compared to the comforts of civilization that you enjoy under Python 2. To wit:

• I cannot find virtualenv for 3.0, which is a disaster. This means that you have to create a separate Python 3.0 install, built with a different --prefix option to ./configure, for each development environment you want to create on your box.
• I cannot find a version of the setuptools available for 3.0. This means limiting your setup.py instructions to the primitive vocabulary of the distutils package. For example, I find myself unable to run the PyEphem test suite at this late hour because I have been running it for so long with:

  $ python setup.py test

  that I am not sure how to get it running otherwise.
• Should you succeed in porting your extension module, it is not at all clear how to distribute it. I had expected either a new PyPI to spring into being — since every package will need an entirely different version under 3.0 — or for a sophisticated scheme to appear for registering one pyephem.tar.gz as the Python 2 version and another pyephem.tar.gz for 3.0. But while the most recent version of your package can mark itself as 2-compatible or 3-compatible (or both) using classifiers, there is no way to have two “most recent” versions of a package. Are we supposed to start distributing a single tar.gz that includes the source code for both Python series, and that selects the right code by detecting the interpreter version at the top of the setup.py file?

So if you make the effort to port your code right now, you might find that the shiny new version of your module is all dressed up, but has no place to go. If you experiment with the following steps, though, you will at least be ready when an official distribution channel does appear for releasing your package into the wilds of 3.0.

Read more

Python 3.0 (r30:67503, Dec  4 2008, 10:23:44)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> [ n*n for n in range(5) ]
[0, 1, 4, 9, 16]
>>>
>>> { n*n for n in range(5) }
{0, 1, 4, 16, 9}
>>>
>>> { n: n*n for n in range(5) }
{0: 0, 1: 1, 2: 4, 3: 9, 4: 16}

Magnificence! Do you feel the waves of beauty crashing over you? No, no, not over the sequence of squares — over the fact that all three basic collection types now support comprehensions!

Comprehensions were first introduced in Python 2.0, but with the terribly awkward stipulation that they were only possible for lists, not for dictionaries. This meant teaching newcomers to the language that list construction was a special case, and that the collections that had deserved their own constructor syntax (lists and dictionaries, at that time; sets came later) were not equally powerful. It also made necessary the awkward and expensive technique of building, and then immediately discarding, a list of tuples to quickly construct dictionaries:

Python 2.5.2 (r252:60911, Nov 14 2008, 19:46:32)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> dict([ (n, n*n) for n in range(5) ])
{0: 0, 1: 1, 2: 4, 3: 9, 4: 16}

The arrival of generator expressions in 2.4 did, at least, allow us to remove the ugly square brackets and avoid creating the list (though all of the tuples still got created then immediately discarded). But the problem remained that dictionaries made from inline generators did not look like dictionaries syntactically.

But, no more! They have even updated the scandalously withdrawn PEP 274 to announce that the feature has finally arrived. After the aching and painful years of the Python 2 series, the language once again shines bright and clear as a model of clever symmetries and low mental impedance. Python's famously tight "feature set" can, now more easily than ever, fit comfortably into the programmer's brain.

What shall I write first in Python 3.0? I wonder.

But you can be sure that my code will find lots of excuses for constructing dictionaries.

The rise and fall of the popularity of “My Shirt” over the four weeks from 5 October through 1 November.

And by drilling down into the list of “Referrers” beneath the graph, I was even able to discover the source of its brief popularity! As Halloween approached, people were doing hundreds of Google Image searches for “waldo costume,” “where's waldo costume,” and “where's waldo shirt,” which brought them straight to my image. As you can see in the above graph, the swelling interest did not peak until Halloween itself, after which the photo plummeted back to its more typical popularity of one or two dozen views per day.

All sorts of gems are hidden in the statistics, waiting to be discovered. For example, it was very satisfying to learn that Yahoo! Image Search considers my wedding photograph A Grandfather in Attendance to be the most important “grandfather” image on the entire web. Behind every statistic is a story about how people find, and why they wind up visiting, each of my photos. Hopefully the fun of watching my old photographs will not distract me from taking some new ones!

Term paper mill. Need $100 by Friday to keep the lights on? No sweat, if you’re a writer. Plenty of kids need ten pages on Hamlet by Thursday... more»

The article, entitled “Term Paper Artist”, alternates between hilarity and poignancy as its author shares his adventures writing hundreds of term papers for hire. But near the end, his tone suddenly becomes serious as he turns to the question of why so many students are unable to write term papers of their own. He thinks that the reason is important enough to stand alone in his article as a one-sentence paragraph. Here is his preceding paragraph, and then the zinger:

It's not that I never felt a little skeevy writing papers. Mostly it was a game, and a way to subsidize my more interesting writing. Also, I've developed a few ideas of my own over the years. I don't have the academic credentials of composition experts, but I doubt many experts spent most of a decade writing between one and five term papers a day on virtually every subject. I know something they don't know; I know why students don't understand thesis statements, argumentative writing, or proper citations.

It's because students have never read term papers.

That is his diagnosis: students never see what term papers are supposed to look like, and so they have no idea how to produce them.

As he continued on, ridiculing the idea that students can produce something of which they are never once shown a good example, I realized that his argument was exactly the same as the one I made in my recent post Reading Code: A Computer Science Curriculum: that the production of any kind of literature, whether an essay in college or an elaborate routine in a computer program, is an essentially imitative act. Without being shown excellent examples from the genre they are expected to produce, students are left in the dark about what, exactly, they are trying to generate — and, more often than not, will fail. They are never even given the opportunity to demonstrate whether they do, in fact, lack the capacity to create, because they are never shown the goal towards which they are supposed to be striving.

I developed the following ideas about how to teach computer programming during a recent conversation with Daniel Rocco, a professor at the University of West Georgia, and Georgia Tech grad student Derek Richardson, and I wanted to expand on the ideas here on my blog. I have been heavily influenced by reading Greg Wilson, and, for all I know, he or someone else may have already put these ideas together into something like the outline below.

The first day of class

To teach computer programming, a professor ought to stride into the room on the semester's first day, explain that a computer program is a text file full of instructions for the computer to follow, and then proceed to check out an example from version control. His desktop should be displayed on a big screen in front of the class. He should read through the example program with them, discuss it — it should be a Python program that does something requiring two or three dozen lines, like opening a window and displaying the message, “Hello, world!” — and then run it. He should demonstrate how to edit the program so that it prints something else instead (perhaps “Hello, professor!”), run the altered version for them, and, finally, check the modified program into version control.

The class now has the tools they need to complete their first assignment. The students are told that a version-control repository has been created for each of them; that they have each been subscribed to their repository's email notification list; and that, upon returning home from class, their email inbox will already hold a message informing them that their first assignment — the “Hello, world!” program itself — has been checked into their repository by the course software. By the next class period they are to have checked the assignment out, altered it so that it prints “Hello” followed by their own name, and submitted the assignment by checking the program back in.

Read more

(Penned after pondering the talk that Dana Gioia gave at Oxbridge 2008)

Because users might become confused now that PyEphem is spread across three web sites — the home page is here at rhodesmill.org, releases are posted over at the Python Package Index, and, again, the development project is now hosted at Launchpad — I have completely redesigned the PyEphem home page with the goal of making the three-site distinction clear, coherent, and easy to navigate. The new home page and documentation are generated by the wonderful Sphinx documentation engine, and I am still thrilled about how pretty my code samples look (check out the one on the PyEphem home page!) now that Sphinx is coloring them in with the renowned Pygments system.

I have simultaneously released a new version of PyEphem that includes the new Sphinx-based documentation, along with several important fixes to the software itself. From now on, rather than cluttering my own blog with every minor version of PyEphem that I might release, fans of the software should visit its News and announcements page on Launchpad and subscribe themselves to its Atom/RSS feed. You will still see the project mentioned here whenever a technical or scientific issue becomes interesting enough for me to write about; but the audience of astronomers and hobbyists who just need to know when the next version is released should not have to wade through my blog to do so!

My users have already begun transferring their questions and problems to Lauchpad, and I look forward to offering much greater accountability through a fully public development process.

United States Constitution

One can spend several minutes just staring at the words so basic to our national life, and pondering the significance of their relative sizes! To make my own contribution to the burgeoning world of Wordle documents, I created a program to extract the memorial messages from the Marshall Booth Guest Book on legacy.com, and then submitted the result to Wordle. After several tries, and after experimenting with the color options, I came up with something I find quite satisfactory:

Marshall Booth memorial

I am sure that Wordle documents will look rather formulaic once everyone has used them to generate Christmas cards one or two years in a row. But they are without question of much greater visual interest than any other tag cloud I have ever seen, and are therefore a big step forward simply by making word frequency something worth staring at.

It would be fun to submit novels, or theological treatises, or each of the books of Paul, to Wordle and then see whether students of English literature or Biblical exegesis could identify the original document simply by which words appeared the most often. I think there would be interesting surprises! Could people tell apart the five acts of Hamlet?

Download slides as PDF

They had asked me to attend so that I could present the Using Grok to Walk Like a Duck talk that I gave at PyCon 2008 back in March. They changed the title, I suppose, to better highlight why it would be of interest to the Plone community; but the change actually helped me to rethink the presentation. I wound up using only the first half of my PyCon slides. For the second half of the talk, which at PyCon had consisted of a crazy sequence of hints and tips about using adapters in your own applications, I instead did a much more successful series of slides about how adapters are actually used in Zope 3 to suit up objects for presentation on the web. I think this made the idea more concrete, and thus much easier to understand for people seeing adapters for the first time.

The talk was well enough received that I should perhaps think seriously about finding further opportunities to share Zope 3 technologies with the Plone community.

I am enjoying my first weeks of using the Tomato Firmware. I purchased a Linksys WRT54GL wireless router because of its admirable support for third-party firmware like Tomato, which replaces the traditional Linksys setup screens with an alterative system with many more configuration options. I can also connect directly to Tomato over SSH and use it as a very small Linux system! This opens endless possibilities for writing fancy firewall rules and running small embedded applications right at the border of my home network.

The Tomato firmware uses a small DNS server named dnsmasq to answer the steady stream of domain name requests from my home computers. It converts domain names that I type, like rhodesmill.org or google.com, into the low-level IP addresses with which computers identify each other.

But I also like using hostnames for the machines sitting right in my home, even though they do not have “real names” out on the Internet. I recommend placing local hostnames inside of a top-level domain that is local to your own network. Choose a suffix that differs from all of the top-level domains that exist out on the Internet — avoid .com, .net, or .uk, for example, in favor of something like .home or .myhouse instead. How, I wondered, could I add extra host names to dnsmasq?

Read more

My efforts were, in the end, successful, and you can see the result — my first two screencasts — in my previous blog entry, which I posted earlier this week.

In the hope that my toil can benefit others, let me outline the details of the process that I have worked out for creating, editing, and posting Linux screencasts. For the impatient, here are the three most important things I learned:

• Linux tools have great difficulty keeping audio and video from gradually going out of sync over several minutes. To avoid problems, always run recordmydesktop with its “on-the-fly encoding” option, and never let “ffmpeg” anywhere near your audio! This not only means that you have to use “mencoder” instead when converting between video formats, but even within mencoder you must avoid the “lavc” audio module, since its code seems to have the same problems.
• Do not attempt to directly edit the resulting Ogg/Theora video! Instead, convert first to the Digial Video (DV) format, and perform your editing there. Be prepared for the fact that DV files are enormous, and that DV pixels are not square.
• Finally, convert to something like AVI for submission to Google Video. Avoid submitting a raw Ogg/Theora file, since even though Google can decode it, their decoders will waste valuable screen space by placing an empty border around the result.

For those interested in more details, I have more to share. Keep reading!

Read more

The audience seemed most interested in the last section of the talk, where I discuss three techiques for debugging problems with Python's import statement; fast-forward to around 3:00 if you want to catch that part by itself.

Next, Jeremy Jones spoke about eggs, Noah Gift introduced virtualenv, and, finally, I got back up to talk about buildout. This was probably my own favorite among the recent presentations I have given, and it's the one I've worked hardest to adapt to a competent screencast:

I have prepared a supplement to the above screencast that gives additional hints and tips about using buildout, as well as a link to the source code of the module that I use as my example.

Finally, if you're ready to see something a little less polished — something that instead of being a screencast is actually a film of me talking in front of a live audience, and gesturing and jumping around — I filled a vacant lightning-talk slot at our February PyAtl meeting with an impromptu introduction to Kinetic Style Sheets (KSS), using an example application that was still sitting on my laptop after at a Georgia Tech developer's luncheon earlier that week:

Now I can finally turn my attention to preparing for my upcoming talk at PyCon 2008 in Chicago! I will be talking about the basic “adapter” design pattern, and how a framework like Zope 3 can facilitate its use. Stay tuned for more information!

PyEphem now also includes a small database of world cities, so that people living near one can simply call ephem.city('Boston') to get their longitude, latitude, and elevation, rather than having to look up the numbers themselves.

I have started using the GraphViz application, which accepts a list of nodes and arrows, and figures out how to attractively arrange them in a diagram. For example, you can very nearly produce this output:

by supplying this rather modest input file to GraphViz (most of whose length comes from my wanting particular colors)::

 digraph Application {
    rankdir=LR;
    node [shape=box,style=filled,fillcolor="#C0D0C0"];
    subgraph clusterClient {
       label="Client"; style=filled; bgcolor="#D0C0A0";
       "Browser";
    };
    subgraph clusterServer {
       label="Server"; style=filled; bgcolor="#D0C0A0";
       "App";
       "Database" [shape=DatabaseShape,peripheries=0];
    };
    "Browser" -> "App" [label="HTTP"];
    "App" -> "Database" [label="SQL"];
 }

I used the words “very nearly” because, in fact, GraphViz only knows how to draw simple shapes like rectangles, and is ignorant of the standard cylinder-shaped database symbol that I have used here by asking for a DatabaseShape. Submitting the above code to GraphViz will, normally, produce three nodes that are all rectangles. To teach it about the database shape, I had to write some PostScript.

Read more

But because I am also really enjoying my work with the new Python web framework Grok, I decided to make it the centerpiece of my presentation

Read more

error: Python was built with version 7.1 of Visual Studio, and extensions need to be built with the same version of the compiler, but it isn't installed.

And, as I myself do not have Visual Studio on the small Windows machine that I deign to own for the sake of my photo printer, I have never been able to offer my users much help. But earlier this year, a helpful PyEphem user named Jeff Kowalczyk emailed me a link to Philip von Weitershausen's post “Cheap binary Windows eggs”, which describes a method for building Python extensions using a freely available compiler.

Read more