A. Jesse Jiryu Davis

Street Retreat Recap

2013-05-23T02:48:54Z

A couple weeks ago I went on my third Zen street retreat, led by Roshi Grover Genro Gauntt. Here's my notes on what happened.

Thursday

We met in Washington Square Park in the afternoon to start the retreat. We had no wallets, cellphones, toothbrushes, books. There were eleven of us. We sat in a circle and meditated. A little jazz combo was playing nearby. We introduced ourselves to each other. Most were on their first retreat, except for Genro, Eileen, me, and Batman. That's his nickname because he was an extra in a Batman movie once. He's a garbage man. He's a long-time student of Bernie Glassman and he's been homeless thirty years. He just got into an apartment this year.

Dinner at the Bowery Mission. We sat through a service first. Indifferent and nearly inaudible Christian guitar rock. We were supposed to sing along, but the lyrics were on a little screen at the front that no one could read, even if we cared to. Much of the congregation was asleep or reading newspapers.

They served beef stroganoff, steamed spinach, nectarines. Fagé yogurt containers were stacked on a table. I stuck with the spinach and a nectarine.

Outside were demolition dumpsters, related to a new New Museum expansion. They were filled with carpet remnants, cut into manageable sizes and conveniently rolled up. All eleven of us chose a roll each and attached them to our bags if possible, or carried them with us. We'd keep them for the rest of the retreat.

We walked to Masjid Farah, the sufi mosque on West Broadway. In past retreats we've wandered looking for it—no cellphones, after all—but Batman knows where everything is and he led us straight there. "Do you want to go the scenic way or the short way?" he asked each time I made a wrong turn.

The streets in the West Village were set up for a big film shoot. Ari and I noticed that we passed right by Fred Armisen standing on a corner talking to a member of the film crew.

The zikr ceremony at the mosque is always a trip. They sing and shout and chant Allah's name until it sounds like gasping. Unlike in past years, we stayed only a couple hours. When everyone got up to dance, we left.

Our plan was to find a place in Battery Park to sleep. We walked with our carpets down to the park. Along the Canyon of Heroes, Jonathan read the name and date of every celebrity ever given a ticker-tape parade in NYC.

We found the park almost entirely fenced off for construction. Weirdly, there was a wide opening in the fence, from which led a fenced corridor to a round, fenced area in the middle of the construction zone. We debated a little: was this illegal? Would we be rousted? Signs on the fence warned of rats and rat poison: were they any concern to us? We bedded down and slept.

Or tried to. I was too cold. The wind was up for much of the night and we were unsheltered. Mike had a T-shirt and his carpet and froze all night. Batman had come with a huge backpack and produced from it a yoga mat and a fancy sleeping bag; he slept well. I had my usual leather jacket and hoodie, and I'd packed a new wool army blanket that I bought for the retreat. I got to sleep a few hours despite the cold. The worst for me was that my blanket shed wool-lint into my eyes, while the carpet from the demolition debris puffed plaster dust up my nose every time I shifted. I was glad to finally hear the birds start to chirp, and to pull the blanket from my face and see the sky brightening.

(Note: Now that I've washed this blanket once, it's not only lint-free but several times softer. Lesson learned.)

Friday

We walked up to McAuley's (New York Rescue Mission) at 6 or so in the morning. Breakfast was a revolting little stack of ham slices and a few pancakes, I think, glued with corn syrup to the bottom of a styrofoam bowl. Strong coffee, though.

We walked to Washington Square Park. Along the way, Batman saw a food cart vendor he knew. He banged on the window and talked to the man a moment, then announced to us: "Anybody want coffee? Tell him how you like it! Step up and get some!" Batman is old friends with everyone.

In Washington Square Park, some sections were soaked, or were being soaked with big sprinklers. We found a dry spot and bedded down for a nap. Still cold, even with the sun up. A parks person came up in a buggy and called out, "Parks! What's going on here?"

Genro: "We're resting."

Parks: "How long are you going to be here?"

Genro: "Maybe an hour."

Parks: "Well okay, we can hold off watering this section for a bit. But just to let you know: water is coming."

Surprisingly accommodating.

Brunch at The Catholic Worker in the East Village, the usual homemade bread and vegetable soup, and good coffee. This is always my favorite meal on retreat. A bottle of hot sauce was on each table: actual flavor!

We walked up to Tompkins Square Park to nap, do meditation and service, and council.

Dinner at the Bowery Mission again. Friday nights they have a pickup jazz band; I've heard it once or twice before, but Friday night they were great. Some big-band flair and wild solos, playing their hearts out, shaking the walls. The drummer, Uros Markovic, has led this band for years. He alternates jazz pieces with bits of preaching. The rest of the band was a sax, a bass trombone, piano, and a woman on the tambourine. I forget her name, but we'd spent time with her on the last retreat, and Batman knows her very well. She seems to have cleaned up, calmed down, put on weight, and become a much better musician since the last time we saw her.

Markovic gave a sermon in which he criticized TV programs that purport to tell us about the real Jesus, that show him as an ordinary man. Jesus wasn't an ordinary man, after all, he was special. We know because the Bible tells us. Muslims get Jesus wrong, too: they don't believe he rose from the dead after three days. Those who know Jesus will live forever in Heaven. Everyone else goes to Hell.

The band did a jazz version of "Nothing but the Blood" by Robert Lowry, which is incredibly weird:

Glory! Glory! This I sing—
Nothing but the blood of Jesus,
All my praise for this I bring—
Nothing but the blood of Jesus.

Dinner was pasta, and some slices of meat that the servers were calling chicken but looked like ham. As is my habit on street retreat, I refused the non-vegetables, so I just had salad. "You want a double portion then? You like avocado? Here, let me put some dressing on. Is this enough? Tell me when! You want some grated cheese?" I love this kind of bustling generosity you find sometimes.

We stood outside the Mission, and a fight broke out between two men, a small black guy and one about my size. The crowd stood back to make room for them. A woman was yelling, egging them on as they assumed boxers' stances and shifted around the sidewalk. Across the street, some men at a bus stop cheered. I thought of intervening but doubted I was the right person for the job. The smaller man threw the first punch, knocked the other down, and raised his foot to stomp on his opponent's head. The man on the ground deflected the stomp with his hands. Batman and a few others rushed in to end the fight. Batman grabbed the winner and pushed him back, arguing with him to leave it be, it's over. He and Batman have been friends for years.

Where should we sleep? Returning to Battery Park would be a long walk in exchange for an uncomfortable night. The wind was up again, and the temperature down.

Batman had two recommendations. The first, an entryway to the Hebrew Union College at West 4th and Mercer. The police treat it like holy ground, Batman said, a refuge: if we stayed just off the sidewalk, in front of the door to the college, we'd be allowed to sleep in plain sight. The alternative was a park in an apartment complex a few more blocks to the south. We found there a big bench-like thing in the middle of the park, a spiral of concrete. It was perfect for 11 people. We walked into the spiral and put down our bedding, cozying up to the shelter of the bench. Residents were passing, walking their dogs, talking on their phones, glancing at us. I wondered what they thought, and if one of them would call the cops on us. And yet, we were out of the wind, and I fell asleep quickly.

Perhaps half an hour later, I heard someone slapping on the concrete. "Hello. Security." I pulled my blanket from my face. A pretty young black woman in a blue nylon jacket was standing outside the spiral. She said that she'd been called with complaints from residents about people sleeping in the park and we needed to move out. If we didn't, a lot more security people would come and it would be a big mess. She talked gently. "I need to stand here until you go." I asked if she knew anywhere else we could sleep. "I'm sorry, I'm not from around here."

We rolled up our carpets and walked a couple blocks to the Hebrew Union College entryway. It's a triangle, a few yards on a side, off the Mercer Street sidewalk. It was enough room for all of us if we slept side by side. Nearby was a small and very warm heating vent, continuously blowing out enough air to warm a half-dozen people standing around it. Batman says that vent is the tribal campfire. He has years of stories of things that have happened around that vent. We warmed up at the vent, then bedded down for the night. It was windier than in the spiral bench, and noisier and brighter, but Batman was right: no one bothered us all night long.

Saturday

We got to the University Community Soup Kitchen, a.k.a. Diane's, a.k.a. the Meatloaf Kitchen, for breakfast at 10am. We sat in rows of plastic chairs and ate open-faced peanut butter and jelly sandwiches on thick slices of white bread, with milky sweet coffee in styrofoam cups. (Homeless services must be the last great consumer of styrofoam in America.) Then we sat on the chairs and waited: lunch wasn't served until 1pm. The regulars who knew each other passed the time talking. Two young white men had a conversation about travel that lasted for hours. Where do you eat in São Paulo? Where do you stay in Bangkok? Once you've bought a ticket to France, how far can you stretch the money you have left once you get there?

An hour into the wait, someone dropped a stack of New York Posts and a couple copies of the New York Times on the table at the front and there was a rush for them. As people finished the papers, they swapped. Eileen talked to someone who'd gotten a Times, and negotiated twenty minutes to read it. We learned Israel had made an airstrike in Syria. Genro and I talked the situation over with a beautiful, chic young black man sitting next to us. As the conversation moved on he told us about the Bowery Residents Committee shelter, on 25th Street. Apparently it's a nice modern-looking place with relaxed rules. He stayed there a few months before "losing his bed," whatever that means. He told a story about sharing a room with a heroin addict and a giant ex-con. The addict came home confused one night. He leaned slowly over the sleeping ex-con, farther and farther, and pissed on him. The ex-con woke up, felt the piss on his face, and beat down the addict. Good times.

A guy I recognized from my last retreat was still there, a big black man with curly hair and a beard. He'd be played by Forrest Whitaker. Last time, he made the lunch wait hard by hunching in his chair and screaming rhythmically all morning. "Give it a rest!" people shouted at him. This time his tic was kind of fun: he'd periodically walk up to someone, touch his throat and sing a few jazzy syllables, nod gravely, and walk away. It was a way to say "hello" to the newcomers: over the course of the morning he seemed particularly to connect with members of our group this way.

Around 1pm the famous meal was served: meatloaf, meatballs in tomato sauce, salad, rolls, coffee. We had real table service. Volunteers walked around refilling our coffee and offering seconds. I make no claim to connoisseurship, but the meatloaf was pretty good.

I was tired of being cold at night, and regretting that I hadn't asked for a coat at the Mission the day before. I asked a volunteer if there was a coat I could have. She was a white woman named Fatima, dressed in an abaya, with a headscarf that covered her piled dreadlocks. On the back of the abaya she had sewn a black Occupy Wall Street patch that covered the width of her back. Her hood left visible a few inches of forehead above her eyes, which was tattooed with green lines in a North African style. She asked whether I wanted a blanket or a coat. What kind of coat would I prefer? I just said, "Whatever's warmest."

Fatima was in the donations room for ten minutes trying to find me the perfect coat. She came out with a green nylon pullover. "Look," she said, "this has a fleece lining, and deep pockets you can put stuff in." She held up the coat and turned it so I could see the lining and the pockets. "You can wear it zipped, or unzipped, and you can tie it around your waist during the day." She demonstrated tying it around her waist. "This should keep you warm tonight." She held up her hand for a high-five.

She gave me the schedule for some homeless advocacy group and insisted I show up. She seemed very concerned about me, and perhaps also she saw in me a possible advocate for the homeless. I was uncomfortable. This had gone on long past the point where I should have explained about the retreat, but it seemed too late to suddenly come clean. This was the only time on the retreat that I misjudged the balance between being honest and blending in.

I rejoined the group with my fancy pullover and we walked up to Tompkins Square Park for naps and council.

Dinner at the Bowery Mission again: this time no music or preaching, and the kitchen moved sluggishly. We are professionals at waiting by now. We sat in the pews for an hour talking to the other guests or listening to their conversations. A man behind me asked his friend to take a look at his eye. "Oh, you definitely have pinkeye. Don't do that! That. Don't rub it, you'll make it spread."

Their conversation moved on to the death of a rapper from Kris Kross. "He was doing coke and heroin together. What do they call that?" "Speedball." "Do you snort it or shoot it?" "Either way, man. Shooting it's dangerous, though."

I forget what was the main course for dinner. I had sickly-sweet iced tea, salad, and an apricot.

We slept in the entryway to Hebrew Union College again. This final night was the coldest yet. I was much better equipped than at the start of the retreat, with my new fleece from Fatima, on top of my leather jacket, and three layers beneath that, but I was still shivering too hard to sleep more than a few hours. But even so: I'd lost any fear of being cold. I was prepared to lie down and shiver until the morning, so that's what I did.

Sunday

I woke up from the cold. I'd learned how the bird songs mark the stages of dawn. When I heard the kind of bird that starts calling around 6, I stopped trying to sleep and got up. There was a party going on around the heating vent: Others had been up since 4. Batman had coffee for us from one of his contacts somewhere—he was acting mysterious about where it came from.

The others had received a box with more cups of coffee, plus cupcakes and a whole cucumber. Some street fair had ended in the late night or early morning, and the organizers had gone around in a van, looking to give their leftovers to some homeless people. Who they found was us.

We stood around the vent talking until 10, drinking coffee, smoking butts we found on the street. What a lovely way to spend a morning.

We ditched our carpets, with gratitude for their service to us, and gratitude that we needn't lug them any farther. We skipped breakfast and had a long and emotional closing council in Washington Square Park.

Wasp's Nest: The Read-Copy-Update Pattern In Python

2013-05-09T02:47:45Z

In recent work on PyMongo, I used a concurrency-control pattern that solves a variety of reader-writer problem without mutexes. It's similar to the read-copy-update technique used extensively in the Linux kernel. I'm dubbing it the Wasp's Nest. Stick with me—by the end of this post you'll know a neat concurrency pattern, and have a good understanding of how PyMongo handles replica set failovers.

Update: In this post's first version I didn't know how close my code is to "ready-copy-update". Robert Moore schooled me in the comments. I also named it "a lock-free concurrency pattern" and Steve Baptiste pointed out that I was using the term wrong. My algorithm merely solves a race condition without adding a mutex, it's not lock-free. I love this about blogging: in exchange for a little humility I get a serious education.

The Mission
The Bugs
Fixing With A Mutex
...and why it's not ideal
The Wasp's Nest Pattern

The Mission

MongoDB is deployed in "replica sets" of identical database servers. A replica set has one primary server and several read-only secondary servers. Over time a replica set's state can change. For example, if the primary's cooling fans fail and it bursts into flames, a secondary takes over as primary a few seconds later. Or a sysadmin can add another server to the set, and once it's synced up it becomes a new secondary.

I help maintain PyMongo, the Python driver for MongoDB. Its MongoReplicaSetClient is charged with connecting to the members of a set and knowing when the set changes state. Replica sets and PyMongo must avoid any single points of failure in the face of unreliable servers and networks—we must never assume any particular members of the set are available.

Consider this very simplified sketch of a MongoReplicaSetClient:

class Member(object):
    """Represents one server in the set."""
    def __init__(self, pool):
        # The connection pool.
        self.pool = pool

class MongoReplicaSetClient(object):
    def __init__(self, seeds):
        self.primary = None
        self.members = {}
        self.refresh()

        # The monitor calls refresh() every 30 sec.
        self.monitor = MonitorThread(self)

    def refresh(self):
        # If we're already connected, use our list of known
        # members. Otherwise use the passed-in list of
        # possible members, the 'seeds'.
        seeds = self.members.keys() or self.seeds

        # Try seeds until first success.
        ismaster_response = None
        for seed in seeds:
            try:
                # The 'ismaster' command gets info
                # about the whole set.
                ismaster_response = call_ismaster(seed)
                break
            except socket.error:
                # Host down / unresolvable, try the next.
                pass

        if not ismaster_response:
            raise ConnectionFailure()

        # Now we can discover the whole replica set.
        for host in ismaster_response['hosts']:
            pool = ConnectionPool(host)
            member = Member(pool)
            self.members[host] = member

        # Remove down members from dict.
        for host in self.members.keys():
            if host not in ismaster_response['hosts']:
                self.members.pop(host)

        self.primary = ismaster_response.get('primary')

    def send_message(self, message):
        # Send an 'insert', 'update', or 'delete'
        # message to the primary.
        if not self.primary:
            self.refresh()

        member = self.members[self.primary]
        pool = member.pool
        try:
            send_message_with_pool(message, pool)
        except socket.error:
            self.primary = None
            raise AutoReconnect()

We don't know which members will be available when our application starts, so we pass a "seed list" of hostnames to the MongoReplicaSetClient. In refresh, the client tries them all until it can connect to one and run the isMaster command, which returns information about all the members in the replica set. The client then makes a connection-pool for each member and records which one is the primary.

Once refresh finishes, the client starts a MonitorThread which calls refresh again every 30 seconds. This ensures that if we add a secondary to the set it will be discovered soon and participate in load-balancing. If a secondary goes down, refresh removes it from self.members. In send_message, if we discover the primary's down, we raise an error and clear self.primary so we'll call refresh the next time send_message runs.

The Bugs

PyMongo 2.1 through 2.5 had two classes of concurrency bugs: race conditions and thundering herds.

The race condition is easy to see. Look at the expression self.members[self.primary] in send_message. If the monitor thread runs refresh and pops a member from self.members while an application thread is executing the dictionary lookup, the latter could get a KeyError. Indeed, that is exactly the bug report we received that prompted my whole investigation and this blog post.

The other bug causes a big waste of effort. Let's say the primary server bursts into flames. The client gets a socket error and clears self.primary. Then a bunch of application threads all call send_message at once. They all find that self.primary is None, and all call refresh. This is a duplication of work that only one thread need do. Depending how many processes and threads we have, it has the potential to create a connection storm in our replica set as a bunch of heavily-loaded applications lurch to the new primary. It also compounds the race condition because many threads are all modifying the shared state. I'm calling this duplicated work a thundering herd problem, although the official definition of thundering herd is a bit different.

Fixing With A Mutex

We know how to fix race conditions: let's add a mutex! We could lock around the whole body of refresh, and lock around the expression self.members[self.primary] in send_message. No thread sees members and primary in a half-updated state.

...and why it's not ideal

This solution has two problems. The first is minor: the slight cost of acquiring and releasing a lock for every message sent to MongoDB, especially since it means only one thread can run that section of send_message at a time. A reader-writer lock alleviates the contention by allowing many threads to run send_message as long as no thread is running refresh, in exchange for greater complexity and cost for the single-threaded case.

The worse problem is the behavior such a mutex would cause in a very heavily multithreaded application. While one thread is running refresh, all threads running send_message will queue on the mutex. If the load is heavy enough our application could fail while waiting for refresh, or could overwhelm MongoDB once they're all simultaneously unblocked. Better under most circumstances for send_message to fail fast, saying "I don't know who the primary is, and I'm not going to wait for refresh to tell me." Failing fast raises more errors but keeps the queues small.

The Wasp's Nest Pattern

There's a better way, one that requires no locks, is less error-prone, and fixes the thundering-herd problem too. Here's what I did for PyMongo 2.5.1, which we'll release next week.

First, all information about the replica set's state is pulled out of MongoReplicaSetClient and put into an RSState object:

class RSState(object):
    def __init__(self, members, primary):
        self.members = members
        self.primary = primary

MongoReplicaSetClient gets one RSState instance that it puts in self.rsstate. This instance is immutable: no thread is allowed to change the contents, only to make a modified copy. So if the primary goes down, refresh doesn't just set primary to None and pop its hostname from the members dict. Instead, it makes a deep copy of the RSState, and updates the copy. Finally, it replaces the old self.rsstate with the new one.

Each of the RSState's attributes must be immutable and cloneable, too, which requires a very different mindset. For example, I'd been tracking each member's ping time using a 5-sample moving average and updating it with a new sample like so:

class Member(object):
    def add_sample(self, ping_time):
        self.samples = self.samples[-4:]
        self.samples.append(ping_time)
        self.avg_ping = sum(self.samples) / len(self.samples)

But if Member is immutable, then adding a sample means cloning the whole Member and updating it. Like this:

class Member(object):
    def clone_with_sample(self, ping_time):
        # Make a new copy of 'samples'
        samples = self.samples[-4:] + [ping_time]
        return Member(samples)

Any method that needs to access self.rsstate more than once must protect itself against the state being replaced concurrently. It has to make a local copy of the reference. So the racy expression in send_message becomes:

rsstate = self.rsstate  # Copy reference.
member = rsstate.members[rsstate.primary]

Since the rsstate cannot be modified by another thread, send_message knows its local reference to the state is safe to read.

A few summers ago I was on a Zen retreat in a rural house. We had paper wasps building nests under the eaves. The wasps make their paper from a combination of chewed-up plant fiber and saliva. The nest hangs from a single skinny petiole. It's precarious, but it seems to protect the nest from ants who want to crawl in and eat the larvae. The queen periodically spreads an ant-repellant secretion around the petiole; its slenderness conserves her ant-repellant, and concentrates it in a small area.

[Source]

I think of the RSState like a wasp's nest: it's an intricate structure hanging off the MongoReplicaSetClient by a single attribute, self.rsstate. The slenderness of the connection protects send_message from race conditions, just as the thin petiole protects the nest from ants.

Since I was fixing the race condition I fixed the thundering herd as well. Only one thread should run refresh after a primary goes down, and all other threads should benefit from its labor. I nominated the monitor to be that one thread:

class MonitorThread(threading.Thread):
    def __init__(self, client):
        threading.Thread.__init__(self)
        self.client = weakref.proxy(client)
        self.event = threading.Event()
        self.refreshed = threading.Event()

    def schedule_refresh(self):
        """Refresh immediately."""
        self.refreshed.clear()
        self.event.set()

    def wait_for_refresh(self, timeout_seconds):
        """Block until refresh completes."""
        self.refreshed.wait(timeout_seconds)

    def run(self):
        while True:
            self.event.wait(timeout=30)
            self.event.clear()

            try:
                try:
                    self.client.refresh()
                finally:
                    self.refreshed.set()
            except AutoReconnect:
                pass
            except:
                # Client was garbage-collected.
                break

(The weakref proxy prevents a reference cycle and lets the thread die when the client is deleted. The weird try-finally syntax is necessary in Python 2.4.)

The monitor normally wakes every 30 seconds to notice changes in the set, like a new secondary being added. If send_message discovers that the primary is gone, it wakes the monitor early by signaling the event it's waiting on:

rsstate = self.rsstate
if not rsstate.primary:
    self.monitor.schedule_refresh()
    raise AutoReconnect()

No matter how many threads call schedule_refresh, the work is only done once.

Any MongoReplicaSetClient method that needs to block on refresh can wait for the "refreshed" event:

rsstate = self.rsstate
if not rsstate.primary:
    self.monitor.schedule_refresh()
    self.monitor.wait_for_refresh(timeout_seconds=5)

# Get the new state.
rsstate = self.rsstate
if not rsstate.primary:
    raise AutoReconnect()

# Proceed normally....

This pattern mitigates the connection storm from a heavily-loaded application discovering that the primary has changed: only the monitor thread goes looking for the new primary. The others can abort or wait.

The wasp's nest pattern is a simple and high-performance solution to some varieties of reader-writer problem. Compared to mutexes it's easy to understand, and most importantly it's easy to program correctly. For further reading see my notes in the source code.

[Source]

Heading Out

2013-05-01T23:04:57Z

I'm heading out for a Zen street retreat tomorrow afternoon. I'll be back Sunday. So if you contact me I'll get back to you next week. And if you meet me on the street, can you spare a dollar?

Another Thing About Python's Threadlocals

2013-04-26T02:24:16Z

As the maintainer of the connection pool for PyMongo, the official MongoDB driver for Python, I've gotten far more intimate knowledge of Python threads than I'd ever wanted.

One of the challenges I face is: if the connect pool assigns a socket to a thread and the thread dies, how do we reclaim the socket for the general pool? I thought I nailed it last year, using a weakref callback to a threadlocal, but there's a bug in that method. Justin Patrin of Idle Games discovered it while testing a PyMongo contribution he's making. I'm going to describe the bug, its impact, the cause, and the fix. I'll conclude by kvetching about supporting archaic versions of Python.

The Bug

Here's some code to start 1000 threads and register to be notified when they're kaput. At the end I assert no thread has died unmourned:

import threading
import weakref

nthreads = 10000
ncallbacks = 0
ncallbacks_lock = threading.Lock()
local = threading.local()
refs = set()

class Vigil(object):
    pass

def run():
    def on_thread_died(ref):
        global ncallbacks
        ncallbacks_lock.acquire()
        ncallbacks += 1
        ncallbacks_lock.release()

    vigil = Vigil()
    local.vigil = vigil
    refs.add(weakref.ref(vigil, on_thread_died))

threads = [threading.Thread(target=run)
           for _ in range(nthreads)]
for t in threads: t.start()
for t in threads: t.join()
getattr(local, 'c', None)  # Trigger cleanup in <= 2.7.0
assert ncallbacks == nthreads, \
    'only %d callbacks run' % ncallbacks

This is the method I presented in "Knowing When A Python Thread Has Died". Each thread creates a "vigil" object and sticks it in a threadlocal. Since only the threadlocal refers to the vigil, the vigil should be destroyed when the thread dies. I make a weakref to the vigil and register a weakref callback. If all goes well, the callback is run as the thread dies. A quirk of Python 2.7.0 or lesser is that the callback is run when the next thread accesses the threadlocal. This oddity is a consequence of Python Issue 1868, fixed by Antoine Pitrou in late 2010 and released in Python 2.7.1.

Note also that I synchronize ncallbacks += 1 with a mutex, since += is not atomic in Python. This innocent-looking mutex harbors a dark intent, as we shall soon discover.

In Python 2.7.1 and newer, the code above works as expected: ncallbacks is equal to 1000 immediately after all the threads are joined. In Python 2.7.0, ncallbacks should be 999 after the threads are joined, and then 1000 after the main thread does the final getattr to trigger cleanup.

The bug is: In Python 2.7.0 and older, ncallbacks is sometimes a few callbacks shy of a thousand. A few threads have been buried in unmarked graves....

Its Impact

I found that an application running Python 2.7.0 or older, if it creates and destroys very large numbers of threads continuously for a long time, and if each thread calls end_request at least once and start_request more times than end_request, will occasionally leave a socket tied to a dead thread. These sockets will eventually exceed the process's ulimit or MongoDB's.

This application pattern would be as weird and unusual as it sounds, which is why no one's complained of the bug.

The Fix

Once I'd written the test code above, I spent a few hours futzing with it—Dammit, I thought this worked! I tried various techniques to force Python 2.7.0 to run the callback a thousand times reliably. Late in the day a divine voice intoned, "synchronize assignment to the threadlocal." So I added a lock:

local_lock = threading.Lock()
# ...
    vigil = Vigil()
    local_lock.acquire()
    local.vigil = vigil
    local_lock.release()
    refs.add(weakref.ref(vigil, on_thread_died))

It worked! Now I was angrier. How can assigning to a threadlocal not be thread-safe?

The Cause

Let's again consider the example code above. The bytecode for assigning vigil to local.vigil is:

28 LOAD_FAST        1 (vigil)
31 LOAD_GLOBAL      3 (local)
34 STORE_ATTR       4 (vigil)

STORE_ATTR calls PyObject_SetAttr, which calls local_setattro, defined in Modules/threadmodule.c:

static int
local_setattro(localobject *self, PyObject *name, PyObject *v)
{
    PyObject *ldict;

    ldict = _ldict(self);
    if (ldict == NULL)
        return -1;

    return PyObject_GenericSetAttr((PyObject *)self, name, v);
}

At the highlighted line it calls _ldict. The _ldict function is, as I've known for some time, a pathetic piece of poo in Python 2.7.0 and older. Here's the turd, edited down a bit:

static PyObject *
_ldict(localobject *self)
{
    PyObject *tdict, *ldict;

    tdict = PyThreadState_GetDict();
    ldict = PyDict_GetItem(tdict, self->key);
    if (ldict == NULL) {
        ldict = PyDict_New(); /* we own ldict */

        PyDict_SetItem(tdict, self->key, ldict);
        Py_DECREF(ldict); /* now ldict is borrowed */
        if (i < 0)
            return NULL;

        Py_CLEAR(self->dict);
        Py_INCREF(ldict);
        self->dict = ldict; /* still borrowed */
    }

    /* The call to tp_init above may have caused
       another thread to run.
       Install our ldict again. */
    if (self->dict != ldict) {
        Py_CLEAR(self->dict);
        Py_INCREF(ldict);
        self->dict = ldict;
    }

    return ldict;
}

We haven't seen any use of the Py_BEGIN_ALLOW_THREADS macro, so one thread's had the GIL the whole time. Locking around the assignment shouldn't have any effect, right?

Well, take a look at the highlighted Py_CLEAR(self->dict) statement—there's the perpetrator. That statement gets the ldict of the last thread that accessed this threadlocal, swaps it with NULL and decrefs it. If this is the last reference to ldict (because the last thread has died) then decref'ing destroys it, and the weakref callback to vigil runs. The callback does ncallbacks_lock.acquire, which releases the GIL before trying to get the mutex.

So here's the kind of scenario I prevented by locking around assignment to the threadlocal:

Thread A starts, assigns to the threadlocal, dies.
Thread A's ldict is now the threadlocal's self->dict and has a refcount of 1.
Thread B starts, begins assigning to the threadlocal, enters the _ldict function.
_ldict sets self->dict to NULL and decrefs Thread A's ldict, which runs on_thread_died, which calls ncallbacks_lock.acquire and releases the GIL.
Now Thread C starts, begins assigning to the threadlocal, enters _ldict.
Thread C finds self->dict is NULL, increfs its own local ldict and assigns it to self->dict. It exits _ldict.
Thread B resumes at Py_CLEAR(self->dict), increfs its own ldict and assigns it to self->dict.

Thread B has now replaced a pointer to Thread C's ldict with a pointer to its own, but it didn't decref Thread C's ldict first. (_ldict wasn't written to survive interruption during Py_CLEAR.) Thread C's ldict will never be destroyed, and a weakref callback to its vigil attribute will never be called.

Locking around assignment to the threadlocal prevents _ldict from running concurrently for any one threadlocal object, and prevents the refleak. In Python 2.7.1 and newer, the whole misguided self->dict system is removed from threadlocals and the lock's not needed.

This scenario applies to PyMongo's connection pool because the pool does need to acquire a lock in its weakref callback. Even if it didn't, there's a possibility of interruption whenever a thread is running Python code.

A Kvetch

This testing, the bug it revealed, the investigation, the fix: all this effort was spent to support entirely obsolete versions of Python. The Python core developers stopped maintaining them years ago, but PyMongo supports all Pythons going back to 2.4, mainly because there are "long-term support" Linux distros like Ubuntu and RHEL that once shipped with them. I have very savvy friends writing new applications on Python 2.6. Our children will have flying cars before we're done debugging these steam-powered versions of Python.

It's particularly frustrating because there's no point even filing bugs against Pythons before 2.7. "We fixed it," the developers will reply. "Upgrade." In Python 2.6, no one can hear you scream.

April Street Portraits

2013-04-21T02:41:19Z

I plan to continue my portrait project at transitional housing facilities. But scheduling those shoots is slow. Meanwhile, I need new pictures for the classes I'm taking, so I photographed some strangers in the East Village.

I notice more than ever, in this set, how much I'm influenced by Hiroh Kikai's Asakusa Portraits. Of course I'm not a fraction of the photographer he is. But like the poet Kenneth Koch said, I "like to be influenced."

Moraff's World

2013-04-20T04:28:02Z

A long-quiescent memory got knocked loose. I recalled that I'd played Moraff's World obsessively as a kid, sneaking out of bed at night to play it on my mother's Tandy 3000. So I downloaded the game and played it for a few hours this week in a DOS emulator.

Moraff's World is a fantasy role-playing game from 1991. It has the usual mechanics of the genre: You choose a race and a class like Fighter or Wizard, explore dungeons, and gain money and items by killing monsters. But Moraff's World is distinguished by its insane complexity. Characters can be one of eight races and seven classes. Killed monsters drop money in seven currencies. There are over a hundred distinct spells, and they come in books, scrolls, wands, and papers. Some characters can learn wizardly or priestly spells, some can learn both. The main UI looks like the bridge of a nuclear submarine:

You look in all four directions at once. There is also an overhead map, like this section of town:

It is the player's job to memorize that yellow squares are temples, red are inns, and so forth.

In a modern role-playing game like Diablo, the town feels alive: music plays, rain patters down, random characters walk around and talk. Moraff's town is vacant and still.

It is not characters who speak to you in Moraff's World, but the programmer Steve Moraff himself. When you enter a bank, the options include PRESS 4 TO ROB BANK. Do so, and the game replies, COME ON! DO YOU REALLY THINK I'D LET YOU ROB MY OWN BANK? PRESS ANY KEY TO CONTINUE.... You are not immersed. You are explicitly in a game designed by a single programmer.

The experience does not resemble a fantasy movie like Lord of the Rings as much as it does reading a fantasy book. When I see a balrog in the movie, I see its fiery skull and its whip, pretty much the same as other viewers. How different from reading Tolkien's description: "a great shadow, in the middle of which was a dark form, of man-shape maybe, yet greater; and a power and terror seemed to be in it and to go before it." I make my own balrog from these words. In the same way, in the absence of any sound or animation, I supply the missing life to Moraff's static world.

This is not to say that the game lacks charm. It is incredibly idiosyncratic. The monsters seem drawn in MS Paint by an exuberant child. Consider this werewolf, and what appears to be a Hawaiian zombie:

The game's engineering is as primitive as its art. Graphics are drawn in layers with the Painter's Algorithm. With each step your character takes, the views in all four directions are re-rendered. The walls are slowly drawn on screen from farthest to nearest. Even on my modern laptop this can take some time when looking down a long hallway. But the technique lends itself to fun effects, like the translucent Shadow Minidragon, partially drawn over the walls behind him:

Moraff applied a similar method to the Wilderness. You may climb a ladder out of town to reach this randomly-generated landscape. It takes several seconds to calculate. (It took several minutes on my mother's Tandy.) If you hit H for Help, Moraff tells you that there's no point exploring the wilderness. It only leads to other dungeons which are all the same. It's a sightseeing expedition.

When I downloaded it this week, my first impression of this Shareware-era game was nostalgic. Back then, games were envisioned by a few people or, in the case of Moraff, one programmer-auteur. There was room for a folk genius to succeed with a very weird game. Nowadays he'd be drowned out by games with hundred-million-dollar budgets like Grand Theft Auto V.

But on second thought, the industry is simply more mature, with a bigger audience and a broader range of games. There's still a place for avant-garde titles developed by small teams, like Braid or Sword & Sworcery. Both use the vocabulary of the simple video games we played as children to evoke grown-up ideas.

Braid is a platform-jumping game, explicitly an homage to Super Mario Brothers. There's even a princess to rescue. But the protagonist has mysterious powers to slow or reverse time, and the game asks: if you had these powers, what would you be? Does the princess want to be rescued? Are you playing the good guy or not?

In the iPad game Sword & Sworcery, the graphics are deliberately archaic and pixellated, but the themes are innovative. The game makes surprising demands regarding its place in the player's life. After you beat a level, for example, Sword & Sworcery pauses for a minute and suggests you take a break and do something else. There are levels that can only be played near a full moon, or a new moon. I changed my iPad's date so I could play them immediately. The designers' goal—to make me aware of my addictive game-love—was accomplished.

The other striking idea of Sword & Sworcery is that one's character does not level up. Instead, with each victory she is weakened. She must keep fighting the same monsters but they grow tougher as the protagonist becomes more vulnerable. At the end, she beats the final boss, but she is retching blood, and flings herself into the river to die. By comparison, role-playing games where your character gains godlike powers seem like childish wish-fulfillment. If you were really a warrior come to save a town, this would be how you'd end up.

The Green Matrix

2013-04-18T21:49:35Z

For a year and a half I've been part of the team maintaining PyMongo, the Python MongoDB driver. It's one of the most widely used Python packages with 1.5 million lifetime downloads. The code itself is only moderately complex; about 8300 source lines. What makes it a tiny horror to work on is the range of environments we support. Here's our test matrix in Jenkins:

That's 72 test configurations. (It looks like more than that, but we don't test Jython and PyPy with C extensions compiled since that currently doesn't make sense.) The dimensions are:

Python version: We support CPython 2.4 through 3.3. On each commit we test just the highlight versions: 2.4, 2.7, and 3.3. We also support the latest Jython and PyPy. We test the intermediate versions like 2.5 and 2.6 before a release.
C extensions: we have a few key parts of PyMongo implemented in C for speed, with pure-Python versions as a fallback. We test both modes.
MongoDB Version: We test the latest development branch of MongoDB (2.5) plus the last two production versions.
MongoDB Configuration: We set up a single server, a master-slave pair, and a three-node replica set, and run mostly the same tests against all.

In each test configuration, PyMongo's test suite has about 430 individual test functions.

This covers the main test matrix, but there are some auxiliary tests we run in Jenkins on every commit. We have a mod_wsgi test that runs a few thousand web requests (first serial, then parallel) against a web app using mod_wsgi in a range of configurations:

Python 2.4, 2.5, 2.6, and 2.7
mod_wsgi 2.8, 3.2, and 3.3
The latest production MongoDB as a single server or replica set

The mod_wsgi tests are there to ensure we never recreate a connection leak like the apocalyptic "unbounded connection growth with Apache mod_wsgi 2.x" bug to which I lost some of the best weeks of my life.

I've also set up some tests for Motor, my non-blocking MongoDB driver for Tornado: I run in Python 2.6, 2.7, and 3.3 against a single MongoDB server and a replica set, running the three most recent versions of MongoDB. I have a separate Motor test that connects to MongoDB over SSL, and finally I have a test of "Synchro," which wraps Motor inside a resynchronization layer and checks it can pass all the same tests as PyMongo. In all, Jenkins runs 33 test configurations for each Motor commit.

Jenkins automatically tests our main configurations, but we periodically hand-test some additional configurations, like sharded clusters, beta releases of Jython and PyPy, and Windows. We'll put some of these in Jenkins too.

For a team of three people to build and maintain this volume of test infrastructure is a huge effort. It's clearly worth it, because the test matrix is so large. But it's not much fun.

Lessons learned:

Test code is a liability: Too much testing code is as bad as too much of any other kind of code. Write as few tests as possible to cover the cases you need to test. Over-testing comforts the novice but impedes agility. For example, when we renamed PyMongo's Connection class to MongoClient, I had to change over 1000 lines in 32 files in the test suite. A commit that huge is a barrier in the repository's history, across which no commit can be moved without conflicts. I hope to never do anything like it again. The test suite should be smaller and better factored.
Tests must be very reliable: It needs to be not only minimal but also very reliable. Tests should fail if and only if the behavior they test breaks. When I joined the team, PyMongo's tests often failed "just cuz." Fixing them all took months: We'd observe an intermittent failure in Jenkins due to some race condition that we couldn't reproduce on our laptops (an EC2 "medium" instance runs a three-node MongoDB cluster slower than you could possibly imagine). We'd think real hard and finally understand and fix the failure. Then we'd do the same for some other test. It was a costly exercise but necessary: It's not until our tests always passed that we took them seriously when they didn't.

There are other dicta that I find negotiable: tests should be fast, sure, but I can live with a test suite that takes a few minutes to run per configuration. Perhaps test methods should include only one assert, but I can live with several asserts in some methods.

I'm implacably opposed to mocking when it comes to testing PyMongo: what our tests verify is primarily our understanding of how to talk to MongoDB. If we mocked any aspect whatsoever of the MongoDB server, our tests would be worse than useless. Virtually every test of PyMongo is an integration test, so we make no distinction between "unit tests" and "integration tests."

I'm curious what others have learned from maintaining a driver's test suite. It seems to be a lot of hard work no matter what.

I Will Pick Up What Others Discard

2013-04-17T18:13:58Z

My friend Jim Roberts emailed me this quote from Master Hua, a founder of Chan Buddhism in the West:

Those in search of the Way should bear this in mind: "I will pick up what others discard." What others do not want, I want; what others will not eat, I will eat; what others will not suffer, I will suffer; what others will not tolerate, I will tolerate; what others will not permit, I will permit; what others will not do, I will do. If you want to support others, you must do it from below. "Seeking the Way from a lower place" means starting from below, not standing up at the top of the mountain. You will never see the Way from the top of Mount Sumeru; but when you are at the very bottom of Mount Sumeru, there you will find the Way.

Jim says this reminds him of street retreat. Yes. Here's my friend Shōin collecting cans during a street retreat last year:

This non-rejecting mind, this mind of spiritual poverty, is the muscle we're training when we're on the street.

DZone Interviews Me

2013-04-17T15:30:29Z

The developer site DZone interviewed me for their "Dev of the Week" feature. I talked about Python, photography, and being teased by my girlfriend. Read the whole thing on DZone.com.

Slides From My Talk On Python Coroutines

2013-04-17T02:27:38Z

Here's the slides from tonight's NYC Python Meetup talk on coroutines in Tornado and Tulip. The slides are a bit inscrutable on their own—it's my style to just show code, then talk a lot to explain the code. Still, if you were there tonight you may find these useful.

Python Coroutines, Present and Future from emptysquare

Toro Rewritten for Tornado 3.0

2013-04-12T20:27:58Z

Speaking of my package Toro, I've just released version 0.5. Toro provides semaphores, queues, and so on, for advanced control flows with Tornado coroutines.

Version 0.5 is a rewrite, motivated by two recent events. First, the release of Tornado 3.0 has introduced a much more convenient coroutine API, and I wanted Toro to support the modern style. Second, I contributed a version of Toro's queues to Tulip, and the queues changed a bit in the process. As much as possible, I updated Toro to match the API of Tulip's locks and queues, for consistency's sake.

In previous versions, most Toro methods had to be wrapped in gen.Task, which made for weird-looking code. But using Toro is now quite graceful. For example, a producer-consumer pair:

q = toro.Queue()

@gen.coroutine
def producer():
    for item in range(5):
        print 'Sending', item
        yield q.put(item)

@gen.coroutine
def consumer():
    while True:
        item = yield q.get()
        print '\t\t', 'Got', item

consumer()
producer()
IOLoop.current().start()

Another nice new feature: Semaphore.acquire and Lock.acquire can be used with the with statement:

lock = toro.Lock()

@gen.coroutine
def f():
   with (yield lock.acquire()):
       print "We're in the lock"

   print "Out of the lock"

More examples are in the docs. Enjoy!

My PyCon Lightning Talk About Toro

2013-04-11T18:29:00Z

The lightning talk I gave at PyCon is now online. I talked for 4½ minutes on Toro, the package I wrote to provide locks, events, conditions, semaphores, and queues for Tornado. Watch for a quick intro on advanced control flow with coroutines:

reStructuredText in PyCharm, Firefox, and Anger

2013-04-10T15:09:07Z

I spend a lot of time writing Python package documentation in reST. Nevertheless, I find reST's markup permanently unlearnable, so I format docs by trial and error: I type a few backticks and colons and angle-brackets and random crap, sphinx-build the docs as HTML, and see if they look okay.

Here's some tools to support this expert workflow.

PyCharm: My favorite Python IDE has basic syntax-highlighting and auto-completion for reST. It's not much, but it far exceeds the amount of reStructuredText syntax that can fit in my tiny brain. It really shines when I'm embedding Python code examples in my docs: PyCharm gives me full IDE support, including automatically adding imports, auto-completing method names and parameters, and nearly all the help I get when editing normal Python files.

There's a file-watcher plugin for PyCharm that seems like a nice way to rebuild docs when the source files change, but it's not yet compatible with the latest version of PyCharm. So instead:

Watchdog: I install the watchdog Python package, which watches files and directories for changes. Watchdog gives me a command-line tool called watchmedo. (I find this fact unlearnable, too; why isn't the tool called watchdog the same as the package?) I tell it to watch my package's files for changes and rebuild the docs whenever I save a file:

watchmedo shell-command --command="sphinx-build doc build" .

Now that I can regenerate HTML automatically, I need a way to reload the browser window automatically:

auto-reload is a Firefox extension that detects any tab with a file:// URL and reloads it when the file changes. In my testing it seems to detect changes in linked files (CSS and Javascript) too. A nice little bar slides down to tell me when it's reloading. That way I know that the reason the page is still a mess is because my reST is still wrong, not because it hasn't reloaded:

This little suite of tools deals well with invoking Sphinx and reloading my web page, so I can focus on the task at hand: trying to write reStructuredText, which is a loathsome afterbirth expelled from the same womb as XML and TeX.

Begging

2013-04-03T18:45:40Z

I periodically spend four days homeless, with a Zen teacher named Genro and a small group of fellow Buddhists. We live, sleep, and meditate on the streets together and eat at soup kitchens. I think the retreat has a triple purpose: First, briefly abandoning the comfort and certainty of my regular life helps me practice non-attachment, the same as it helped the first Buddhist monks. Second, it gives me a taste of what it's like to be homeless, so I can better understand the homeless people I meet in NYC. And finally, it's an opportunity to raise money for homeless services.

The rule is that I must raise $500 by May 2. The money will be distributed among the organizations that help us while we're on the street, and it will support the social service activities of the Hudson River Peacemaker Center. I have to beg for the money—I'm not allowed to just donate $500 of my own.

So I'm begging you: Will you please donate?

Update: I've now (April 16) exceeded my minimum, with $803. But donate anyway! Additional funds are divided the same as the first $500.

Shuso Hossen, Spring 2013

2013-04-03T02:53:38Z

Two weeks ago the Village Zendo completed a week-long urban sesshin focused on our awareness of disabilities. We were blindfolded for part of one day, and wore earplugs for part of another. The retreat ended with the Shuso Hossen ceremony, in which R. Liam Oshin Jennings gave his first dharma talk.

Review of "MongoDB Applied Design Patterns" by Rick Copeland

2013-03-26T19:28:03Z

There's a lot of bad advice out there regarding MongoDB. As I wrote in my last review, even smart sources can encourage risky methods. Soon, I hope, there will be as much good MongoDB instruction from experts outside 10gen as there is good third-party SQL instruction. For now, know that you can trust Rick Copeland.

Copeland's new O'Reilly book on MongoDB complements O'Reilly's other five: the majestic Definitive Guide (due for a second edition in June), Scaling MongoDB, 50 Tips and Tricks, and the MongoDB books for Python and PHP.

After you've read the Definitive Guide, a good candidate for your second MongoDB book is Applied Design Patterns. (Disclosure: I was paid to critique an early draft.) Copeland's intended audience has basic MongoDB competence and wants application examples that optimize either for scalability or maintainability, plus the principles to guide new designs. Copeland also assumes basic SQL knowledge, and presents most examples in contrast to conventional SQL solutions, a method I find distracting and irrelevant. He identifies some common application types (product catalog, CMS, analytics, etc.) and provides for each a schema and application logic. He goes far beyond prior works when he discusses performance, consistency guarantees, and sharding considerations for every application.

In Part 1, Copeland discusses the basic questions about MongoDB schemas. Right away, he identifies what makes nonrelational design different:

There is no longer a "garden path" of normalized database design to go down, and the go-to answer when faced with general schema design problems in MongoDB is "it depends".

MongoDB requires optimization up front, more often than SQL schema design does. (Armin Ronacher noticed this too a few months ago.) Most often the question is whether to embed or to link, and what data should be normalized or denormalized. Copeland uses an extensive description of disk seek times to explain the motivations for embedding and denormalization, better than prior MongoDB schema-design materials have.

Many presentations, my own included, have claimed that you can migrate your schema lazily with MongoDB: your application can start writing data in a new format, and read data in both new and old formats, while a batch job slowly migrates old data. MongoDB Applied Design Patterns finally presents a complete example of lazy migration, including example code (in Python) for reading data in both formats while the migration is in progress.

Without general-purpose transactions, MongoDB requires new techniques to guarantee that a series of changes is atomic: that is, to guarantee that in the long run your data either reflects all the changes or none of them. The simple approach is to put all related data in one document and use update operators to modify all the data in one shot. If there's no way to restrict your atomic operation to one document, your next best bet is optimistic concurrency control: try to complete the operation, check if another process overwrote your changes, and if so retry them. There are a number of examples of this in the wild (the MongoDB Manual, Dan Crosta, Scott Hernandez); Copeland's contribution is unusually complete, with example code for handling every case that can arise.

Part 2 of the book is much longer, and covers six kinds of application in depth, both conventional (a social network) and unusual (a role-playing game). Here Copeland excels. Where he covers well-tread ground his designs are more detailed and better thought out than prior authors', and where he innovates he chooses interesting problems to solve. In the Operational Intelligence chapter he explains compound indexes clearly and correctly. He presents a complete design for an analytics application using the MongoDB aggregation framework, and covers the interactions between aggregation, indexes, and sharding.

The final example of the book is an online Zork-style game. This is less widely applicable than E-Commerce or content management, but way more fun. Copeland chooses to radically denormalize his schema: when a player enters a room, the room's entire data structure is copied into the player's document so the game can display the player's state without querying for the room again. As with the other examples, this application is considered in depth: each query is carefully indexed, and when a player picks up an item, Copeland's code prevents another player from picking it up concurrently. Most of the game's intelligence is expressed in Python code rather than in MongoDB queries. Developers using Oracle or Microsoft SQL Server tend to push all the logic and complexity into their schema, their queries, and stored procedures. With MongoDB's simpler feature-set, coders have to move more logic out of the database and into their application. If a SQL refugee hasn't yet learned this lesson, the gaming chapter will drive it home.

Review of "Building Node Applications with MongoDB and Backbone"

2013-03-23T16:42:26Z

Mike Wilson's O'Reilly book from December 2012 introduces some hip web development techniques by building a book-long example of a social networking app. Besides introducing MongoDB, Backbone, and Node, he shows the beauty and remarkable concision of Jade, Require.js, and Mongoose. He demonstrates good patterns for organizing your code in an application of substantial complexity, covers a lot of ground in few pages, and concludes with an unusually feature-complete chat-server example that weaves together all the layers of the stack. Wilson has some dangerous habits readers shouldn't emulate, but on balance his book teaches well.

By necessity, the book jumps frequently between Node and Backbone, models and views, HTML and Javascript. It's the nature of web development that each new feature requires changes in many places, and it's hard to stay oriented. Wilson maintains a corrected version of each chapter's code on Github; use that instead of relying entirely on the examples in the book.

I've built one large front-end Javascript application with Backbone, and I floundered at organizing it. Although Backbone is rigorous (hence the name) about separating models and views, higher-level questions are underspecified: how should the code be split among files? Whose responsibility is it to create the models and views? Wilson uses Require.js to neatly slice code into files and to declare the dependencies among them. In his example application, the Backbone router is responsible for instantiating all models and views. As the book progresses and his example application grows, the routes, models, and views remain focused and decoupled. It's a compelling design. I wish I'd known.

Wilson spends an early chapter building a login system for his example app, before implementing any features. He even salts his password hashes to defend against rainbow tables. An author less secure in his convictions would fear losing his reader's attention, but Wilson insists on doing the right thing. And rightly so: readers will paste his examples and put them into production, so the examples should be complete.

On the other hand, Wilson's introduction to MongoDB misses some marks. It's only 12 pages, so why did he spend two of them on MapReduce? MapReduce has always been intended for big batch processes, not web applications. MongoDB books and talks have long over-emphasized MapReduce, which should be confined to a niche. The aggregation framework, on the other hand, is general-purpose and was released months before Wilson's book; it should have been covered instead.

Wilson also shows a MongoDB pattern that risks losing updates and is needlessly slow: When a user adds a contact in his social-networking site, Wilson's code fetches the whole user document, adds the contact, and saves the whole document back:

app.post('/accounts/:id/contact', function(req,res) {
  var accountId = req.params.id;
  var contactId = req.param('contactId', null);

  models.Account.findById(accountId, function(account) {
    models.Account.findById(contactId, function(contact) {
      models.Account.addContact(account, contact);
      account.save();
    });
  });

  // Note: Not in callback - this endpoint returns immediately and
  // processes in the background
  res.send(200);
});

(I've edited for brevity; the whole code is on GitHub.) Note that if two requests are updating the same account, the first one's updates are lost. $addToSet would have solved this, and would be more efficient too.

Equally worrisome is Wilson's tendency to drop errors on the floor instead of reporting them to the user, as shown at the bottom of this function. He argues "we are accepting the small but rare inconvenience in order to serve the majority of requests at an accelerated speed." This is a terrible argument for silencing errors, especially since the front-end framework needn't block the user from interacting with the UI while it waits for the server response.

A book like this seems intended to show best practices, and patterns that encourage correctness. Some of the hardest patterns to learn are error-handling in Node and concurrency control in MongoDB. I wish Wilson had devoted half the attention he placed on security to these two topics.

But I'm only mad at these flaws because the book they mar is a good one. As Wilson builds up his architecture piece by piece, the patterns appear both usable and elegant, and capable of staying clean as the app grows. Wilson uses Backbone custom named events like "app:loggedin" or "chat:start" to coordinate his front-end code, instead of letting views directly call methods on other views. A novice Backbone user might not see the tremendous value of decoupling views this way, but take it from me—it's a great idea.

The book concludes with a long chat example. Chat examples with Socket.io and Node are legion—indeed, obligatory—but the completeness of this one, including its integration with Backbone, is a tour de force. If you plan to use either Node or Backbone this book has excellent recommendations for structuring a large app, and even if you're not building with any of the frameworks Wilson covers, his examples can inspire you to write more concise and decoupled code.

Slides from my PyCon lightning talk on Toro

2013-03-16T18:48:21Z

Here's the 8 slides for my 4½-minute talk on Toro this morning. Toro is a package I wrote last year that provides objects something like locks, events, conditions, semaphores, and queues for Tornado coroutines.

PyCon lightning talk on my Toro module for Tornado from emptysquare

Plop: Python Profiler With Call Graphs

2013-03-11T19:37:35Z

Tornado's maintainer Ben Darnell released a Python Low-Overhead Profiler or "Plop" last year, and I'm just now playing with it. Unlike cProfile, which records every function call at great cost to the running process, Plop promises that "profile collection can be turned on and off in a live process with minimal performance impact."

A Plop Collector samples the process's call stack periodically (every 10 milliseconds by default) until you call Collector.stop(). Plop's profile viewer is a web application built on Tornado and d3.js, which uses a fun force-directed layout to display your process's call graph. You can use the demo scripts from Plop's repo to make an example profile:

Functions are shown as circles, sized according to the number of times they were executed and colored according to filename. Edges connect callers to callees. The visualization nearly freezes Firefox but runs well in Chrome.

Plop isn't going to replace cProfile and RunSnakeRun, but that's not its intention. Better to think of it as a lightweight complement to the heavier machinery: Plop is nice for visualizing call graphs (which RunSnakeRun does badly) and for sampling a live process in a performance-critical environment.

Review of Roman Vishniac Rediscovered

2013-03-11T03:15:45Z

Today I saw the International Center of Photography's big retrospective, "Roman Vishniac Rediscovered". The show opens with Vishniac's Berlin street photography from the 1920s and 30s, in which he concentrates on form: shafts of light in a train station; a workman on a diagonal ladder amid diagonal shadows; four boys admiring a motorcycle, all dressed alike. The beauty and the visual coincidences he catches are delightful. The scene darkens as the Nazis rise to power, and the impact of the photos, unfortunately, wanes. Vishniac's photo of his daughter wearing a cute beret, standing in front of a Hitler poster, is ominous, but not particularly good.

Vishniac's most prominent achievement is his photographs of Eastern European Jews in the late 1930s. The project was commissioned by an American Jewish relief fund to highlight the poverty of Jews in Eastern Europe, much in the same way (and at the same time) as the FSA commissioned Dorothea Lange and Walker Evans to photograph the Dust Bowl. ICP displays the work in fine new inkjet prints from Vishniac's negatives, and sometimes shows images Vishniac had originally edited out: Jewish women in secular dress, for example, or a prosperous-looking Jewish shop. The exhibit demonstrates how Vishniac selected his photos to accomplish a narrow view of Jewish life: poor, religious, medieval. When this world was wiped out by the Nazis a few years later, Vishniac's record of it became a twilit elegy, but the work as we've known it is not the whole scene Vishniac saw.

Propagandistic, too, are Vishniac's 1939 photographs of a Dutch "agrarian training camp" that prepared Zionist youth for emigration to Palestine. The images are posed, with clear inspiration from Socialist Realism. They're of their time: the age of statism, when individuals everywhere were subsumed in one ideology or another.

It makes one nostalgic for the pictures made before all the polemics, when Vishniac was satisfied just to photograph stylish figures in slashing light. Unburdened by any message, these images are light, and the best in the show.