Concision and Concinnity

ouimeaux 0.5.1 released: Fuzzy device matching, bug fixes

2013-11-09T11:01:00.003-05:00

Several new features making it easier to use, as well as a lot of bug fixes. Full list here. Included:

wemo status/wemo switch status : Prints the current on/off status of all switches or specific switches in your environment.
Fuzzy device name matching: You can now use fuzzy matching on the command line so you don't have to preconfigure aliases. For example, wemo switch tvrm on will work as well as wemo switch "TV Room" on .

The command-line help is also much more complete. Check it out!
ouimeaux on GitHub
ouimeaux on PyPI

ouimeaux 0.3 released: Device Caching, Aliases, Configuration Options and More

2013-05-25T15:51:00.003-04:00

In response to a couple of tickets, I've released ouimeaux 0.3. It includes some bug fixes and a few nice features:

Device Cache

Discovery results are now cached on the filesystem, so command-line operations will be much faster to initialize. The cache can be disabled with the --no-cache option.

Aliases

Some device names are long or contain spaces, which makes using them on the command line unwieldy. You can now define aliases that make that easier to deal with. For example, you can alias "TV Room" to "tv" allowing you to run wemo switch tv on.

Discovery Response Server Binding

The UDP server that listens for responses to the UPnP broadcast can now be configured to bind to a specific IP and port.

Config File

You can configure all of the above by modifying ~/.wemo/config.yml. It's fairly self-explanatory.

Toggle command

Tiny feature: You can now pass "toggle" as well as "on" or "off" as a wemo switch command.

pip install or fork at will, and let me know how it goes.

PyPI

GitHub

ouimeaux 0.2 released: Now with Motion events!

2013-04-22T10:15:00.002-04:00

Finally had a chance to fire up Wireshark and figure out how WeMo Motions and Switches talk to each other. I've worked the results of my discovery into ouimeaux 0.2. Now you can register callbacks to be notified whenever a Motion or Switch changes state (i.e., when motion is detected or a switch is switched). I also cleaned up a few bugs with the UDP server, so you should be able to discover periodically now without it throwing an error.

https://pypi.python.org/pypi/ouimeaux/0.2
https://github.com/iancmcc/ouimeaux

Controlling Belkin WeMo switch with a Raspberry Pi

2013-02-07T01:05:00.001-05:00

Here's a simple example of a way to use the ouimeaux library. The RPi's running a server that discovers a WeMo switch by name (specified on the command line), then essentially forwards switch input from the GPIO pin to the WeMo. With a power supply circuit to let it run off 110 AC and a nicer switch, you could mount this in the wall to control your WeMo a little quicker than pulling out your phone, for those cases when you're already in the room. Said my sister: "Congratulations, you've reinvented the light switch."

The switch circuit is the simplest possible; example circuitry and code can be found here.

ouimeaux: Command line and Python API for WeMo hacking

2013-02-07T01:03:00.002-05:00

I bought a bunch of Belkin WeMo Switches over Christmas, and while I appreciate that my lights now have IP addresses, controlling them can really only be done through the WeMo iPhone app, which (besides having a few understandable bugs) is slow to start up. More to the point, I had no way to hook the switches up to any other events beyond using IFTTT, which seems to have a one-switch limit.

Luckily, WeMo devices appear to provide a fairly complete SOAP API. Isaac Kelly posted a proof of concept discovering and controlling switches, so obviously more direct control was possible. Hacking together a decent library only took a couple of nights.

Thus: ouimeaux. It uses gevent for async I/O and the minimal UPnP functionality required (broadcasting M-SEARCH requests, listening for responses), and it uses requests to communicate directly with the devices. Service stubs are built on the fly and provide easy attribute access to actions (for example, switch.basicevent.SetBinaryState(BinaryState=1)). There's also a handy explain() method to print out all services, actions and arguments. The only useful things I had time to figure out how to do—turn switches on and off, get current switch state—I pulled up into methods on the switch object itself: switch.on(), switch.off(), switch.get_state(). The documentation gives more details.

Once that was written, a simple command line script was straightforward. Installing the package will get you the wemo script in your PATH. wemo list to discover names of devices, wemo switch "Switch Name" on to turn it on. Nothing else so far, but it's highly hackable.

I hope to explore the Motion API next, and provide the ability to listen to events from Motion devices. Firmware uploading is exposed too; might be possible to intercept and modify to get an SSH server running.

pyxdeco: Python eXtraordinary Decorators

2011-01-31T09:59:00.000-05:00

Decorators in Python are limited to action when the function they wrap is actually called. Which is fine and all, but they can be more useful. Or at least could be, if they allowed for hooks at definition or instance creation. I've packaged up a technique I've been using to do just that and released it as pyxdeco.

I wrote the code when trying to register instance methods as event listeners automatically when the instance was created. Decorator syntax made the most sense by far:

...
@on('click')
def click_listener(self):
    ...

...but Python decorators can't be used that way, since the decorator wouldn't be invoked until click_listener was actually called.

This package adds the ability to define decorators to be called at any of the three points of the function's existence: when the function is defined, when an instance of the class containing the function is created, and when the function is itself called (the last is of course not new, just a regular decorator, but is provided for consistency and completeness).

Code is on github, downloads available from pypi. Suggestions for examples or future development welcome.

Decorator Factories (and decorator factory factories)

2010-09-08T09:30:00.000-04:00

Decorators are a simple, handy way of implementing certain features in any language, but many people find them perplexing. When you start talking about functions that generate decorators, it gets a little twisty; when you then make helpful functions to create functions that create decorators, you might have to stop for a minute to unclench your brain. Really, though, the concept is quite simple if you take it step by step, so here it is (after the jump), hopefully demystified.

Decorators

For the uninitiated, a decorator is a function that replaces another function, adding extra functionality along the way. Here is a trivial example in Python:

def squared(f):
    def inner(*args):
        result = f(*args)
        return result**2
    return inner

def timestwo(n):
    return n*2

timestwo = squared(timestwo)

squared() accepts a function and returns a function, which is the fundamental characteristic of a decorator. We then decorate timestwo() by passing it through the decorator and replacing the reference to the original function with the result. After all this has executed, timestwo is now actually the closure inner, which keeps a reference to the original timestwo in the decorator's scope so it can call it and mess with the result.

In Python you can use nice decorator syntax:

@squared
def timestwo(n):
    ...

Which is synonymous with the manual call to the decorator and reassignment.

Decorator Factories

So let's say you want a more flexible decorator that can raise to any power, not just 2. We could individually define squared(), cubed(), timed() and so on, but it's much easier just to create a function that returns the decorator we need. We'll make a decorator factory, raisetopower, that returns a decorator, decorator, that returns a function, inner, which will replace the decorated function. Again, we'll use closures all the way down, so we keep all the information we need in scope:

def raisetopower(exponent):
    def decorator(f):
        def inner(*args):
            result = f(*args)
            return result**exponent
        return inner
    return decorator

Since raisetopower(2) will return what is essentially squared() from above, we could do:

squared = raisetopower(2)
cubed = raisetopower(3)

But it is much easier just to do:

@raisetopower(2)
def timestwo(n):
    return n*2

This may look strange to some, but bear in mind that it is functionally identical to:

timestwo = raisetopower(2)(timestwo)

And since raisetopower(2)==squared, we've recreated the same situation we had above, only we can now pass in any exponent we want.

Decorator Factory Factories

Naturally the same technique, creating and returning a closure wrapping the original function, can be nested indefinitely; at some point it becomes unreadable. Personally, I would never have multiple calls on the same line, so I draw the line at actually calling decorator factories. I have, however, found uses for decorator factory factories, which you would need when you wanted a reusable decorator factory with some information frozen inside. For example, in txrestapi, I needed decorators that tied a function to both a request method and a regular expression, like:

@GET('^/.*$')
def onGetAnything(self):
    ...

GET itself is just a decorator factory, but I defined GET (and POST, and PUT, and DELETE) using a decorator factory factory:

def decorator_factory_factory(method):
    def decorator_factory(regex):
        def decorator(f):
            def inner(*args, **kwargs):
                # do stuff with f(*args, **kwargs)
                # and method and regex
            return inner
        return decorator
    return decorator_factory

GET = decorator_factory_factory('GET')
PUT = decorator_factory_factory('PUT')

Now, I could have used the function to do:

@decorator_factory_factory('GET')('^/.*$')
def onGetAnything(self):
    ...

But as I said above, you reach a point where it's totally unreadable, so draw the line appropriately.

So that's that: decorators, and functions that create decorators, and functions that create functions that create decorators, all unpacked. Really it's just the same technique, replacing a function with a closure wrapping that function, over and over.

Released: txrestapi 0.1

2010-09-06T15:02:00.001-04:00

Sorry updates have been horribly infrequent; I both had a kid and moved to Austin (in that order) in the last few months. With that out of the way, to the code...

I got frustrated chaining twisted.web.resource.Resource classes to provide a complex REST API service, so I wrote a new package that lets you do it with regular expressions and decorators, like:

@GET('^/path/to/api/method$')
    def on_api_method(self, request):
       ...

I'm pretty fond of it already. Code is on GitHub, package is available from pypi — and, of course, via easy_install. Full docs are at both sites, quick example after the jump.

Some quick tips:

If you give your regular expressions named groups, those will be passed into the callback as keyword arguments (like Django).
You can use the ALL decorator to register a callback for all methods, instead of just GET or POST or PUT or DELETE.
You can use multiple decorators on the same callback to register the same callback for exactly those methods.
Callbacks are evaluated in order, top to bottom, so put the most specific at the top.
You can return Resource objects from a callback as well as strings, at which point normal twisted.web traversal takes over. You can also, of course, return other APIResource objects.

from txrestapi.resource import APIResource
from txrestapi.methods import GET, POST, PUT, DELETE, ALL

class MyAPI(APIResource):

    @GET('^/(?P<id>[a-zA-Z0-9]+)')
    def get_info(self, request, id):
        return "Requested information for id %s" % id

    @POST('^/(?P<id>[a-zA-Z0-9]+)')
    def set_info(self, request, id):
        return "Setting properties on id %s" % id

    @PUT('^/(?P<id>[a-zA-Z0-9]+)')
    def create_ob(self, request, id):
        return "Trying to create object with id %s" % id

    @DELETE('^/(?P<id>[a-zA-Z0-9]+)')
    def del_ob(self, request, id):
        return "Deleting object %s" % id

    @ALL('^/')
    def default(self, request):
        return "I match everything; clearly, you aren't asking for anything interesting."

from twisted.web.server import Site
from twisted.internet import reactor
site = Site(MyAPI(), timeout=None)
reactor.listenTCP(site, port=8090)
reactor.run()

Then make assorted requests to http://localhost:8090/someid and watch them get routed to your callbacks.

Notes on Teaching JavaScript

2010-03-02T22:57:00.001-05:00

For the past few months, I've been tutoring a friend in JavaScript, and I've rediscovered some trite knowledge anew: there's no better way to improve in a subject than to teach it.

My friend and I come at programming JS from different directions: he's primarily a Web designer who uses jQuery to enhance pages that can stand on their own, while I'm a Python/Zope developer who's currently using ExtJS as the framework for a full-blown web application. I did a lot of thinking before our first session about how to approach the undertaking—should I teach to a framework, since that would yield the most immediate benefits? If so, which? Would I be able to explain best practices without going laboriously over never-again-used fundamentals of programming? Would our sessions even be worthwhile, since hundreds of tutorials are freely available?

Luckily my tutee was enthusiastic about learning from the bottom up, which was significantly easier for me, since that's how I learned myself. Despite the familiar trajectory, our sessions have been pretty educational for me, too, in part because they've highlighted the difficulties (and rewards) of teaching JavaScript, as distinct from those involved in learning it.

The first thing I realized was almost a tautology: different people approach JavaScript in different ways. The gap between designer and developer is quite large, and yet their disparate approaches are at once equally valid (grant me the possibly false dichotomy for just a minute). Other things being equal, designers want to add features and effects to their pages, to enhance the user experience. They think in CSS, and are primarily concerned with the manipulation of DOM elements. Developers, on the other hand, are used to complex APIs and constant refactoring, and are concerned with abstraction and (hopefully) readability.

A designer may be perfectly happy with a long jQuery chain that would cause a developer spiritual distress; a developer may confront a simple problem with an elegant but unnecessarily complex solution that only serves to frustrate the designer. Being primarily a developer myself, I had to realign a bit, to overcome my instinct to make things as useful as possible, instead of merely as useful as necessary. At the same time, I felt it was important to present the basic programming mores that guide the more technically minded scripter—jQuery chains be damned.

The second thing I realized: It's easy to learn JavaScript wrong. The overhead of fooling around with JavaScript is incredibly low. Everybody uses a browser every day, and as soon as you learn how to open the console, or install Firebug or Greasemonkey, you can have a visible effect on any of the hundreds of sites of whose shortcomings frequent use has made you achingly aware. At the same time, decent entry-level JavaScript information is oddly hard to come by. Most you find on the Web is outdated. Few tutorials adequately address cross-browser issues. And when you finally give up in frustration, there's always jQuery, which is incredibly easy both to use and to misuse.

Had jQuery been around when I needed the simple solutions, I might easily have managed to avoid the dirty low-level stuff to this day. Nothing, however, teaches you JavaScript quite so well as rolling your own cross-browser Ajax utilities, and so I studied the source code of Prototype and script.aculo.us and MochiKit, and now I'm a far better programmer than someone starting out today might be forced to become.

Since education was, in the instant case, the desired goal, I shied away from, say, teaching to jQuery (or even ExtJS or YUI, which are much closer in coding style to JavaScript per se), and guided our sessions as much as made sense towards a home-grown solution. Since my tutee is partially responsible for a large public site that needs only moderate, self-contained effects and widgets, but also would like his code to be reusable and readable by his many colleagues, he needed a little of the approach of both the designer and the developer. And so our ongoing project became to write a reusable, widget-focused library.

Creating a simple library turned out to be a fantastic learning exercise: it requires investigation of namespaces, presents plenty of scoping problems to overcome, and requires attention to consistency, organization and sane API design. It's immediately useful, so no worries about the student getting bored. A library is infinitely extensible, but can also contain the most trivial low-level utilities. Every library has its own style, and lets its coders be clever.

Most importantly, the creation of a library invites, and even insists upon, reiteration of what I find to be the most helpful guideline to keep in one's head while writing code: Assume a third party will one day attempt to extend everything you produce.

Personally, I've found the experience quite valuable, for one crucial reason: I'm not a designer. The interfaces and APIs I'd normally create would probably be most useful to somebody like me. Watching the thought processes of somebody in the thick of it by day as they design an API they'd like to use is enormously instructive.

Now, we're using jQuery as the basis for this library, because even though cross-browser issues are still unfortunately relevant, let's face it: frameworks are here, they aren't going anywhere, and they solve the problem without much effort, so it's not a paramount concern. Nonetheless, whenever we run into a problem, we write a solution from scratch before we use a jQuery shortcut (for example, we wrote a scope-setting decorator function similar to jQuery.proxy rather than just using proxy). When we've found a need to use something more complex, like class-like inheritance, we've studied many variations and written our own, and finally settled on a style that feels like jQuery but adds substantial value (no small feat, as the concept of class inheritance is nearly diametrically opposed to the jQuery ethos).

Probably nobody but my tutee's band of designers will ever use this library. But it's a great propadaeutic compromise, for both of us. I have an opportunity to share the best practices I've managed to distill from the pored-over source of other frameworks, and I get the chance to see things from the designer's point of view, rather than focusing on that of the implementer (me) or end user (actual end user).

If you can find a willing subject, I'd recommend the practice of teaching JavaScript as an excellent way to coalesce the lore into a science.

Personal Agile Task Management with OmniFocus

2010-02-28T00:08:00.000-05:00

The Zenoss dev team uses Rally for project management. It's fine for planning the activities of the entire team, but as a personal task manager I find it pretty useless. Its interface is unwieldy, and it doesn't allow for more or less fine-grained tasks. Great to get a high-level view of an iteration; bad for any low-level organization.

Since I like OmniFocus for my own task management, I've developed a system that mirrors Rally at the highest level, but allows for much more versatility at a lower level (not, of course, restricted to Rally; should function perfectly well for any Agile project). OmniFocus and Rally sync up surprisingly well, as far as that goes, but OmniFocus lets me take those extra steps to get myself an actual task list for the day and switch priorities on an hourly basis.

First, the basic translations:

Agile Concept	OmniFocus Analogue
Iteration	Project Folder
User Story	Project
Task	Action
Estimate	Estimate

Now, the details. I've organized my project folders at the top level according to my various roles; under "Zenoss Developer" I have two folders, Current Iteration and Backlog. Since I don't use OmniFocus for project planning, just for task management in the foreseeable future (which usually doesn't include next iteration), there's no need to have folders for every iteration in the release.

When I break a project down, I put the resulting projects and tasks under the Backlog folder, in subfolders according to feature ("User Interface," etc.). I also have a single-action list called Tickets into which go actions representing tickets I need to fix. I try to include time estimates where appropriate.

After we finish our iteration planning, I modify the projects in the Backlog folder with any changes, then I drag those projects and tickets planned for the iteration into the Current Iteration folder. I then break down tasks into sub-tasks for the nitty-gritty stuff that's too small for Rally but for which I still want a reminder.

When I create a project based on a user story, I use a template prepopulated with the tasks associated with every user story—code review, running unit tests, communication with QA, etc. To this I add the tasks unique to the particular project. This cuts down on the drudgery of data entry while ensuring I don't forget anything. Contexts are all preset as well.

Now my iteration is easy to see (and manage) at a glance. I have a "Current Iteration" context focused on the project folder, so I'm not distracted or overwhelmed by the entire project at once. When priorities shift, I can change the order of the projects within the Current Iteration folder, or move things in and out of the Backlog folder.

When the iteration nears its end, I've already begun to do some of the planning needed for the next iteration simply by virtue of using OmniFocus in a way compatible with Rally. Anything left in Current Iteration gets split to the next iteration, and whatever's at the top of Backlog gets dragged into Current Iteration to fill in.

I don't worry too much about time estimates in OmniFocus, although I do make some attempt to populate the field just for the sake of keeping the information at hand. But generally I leave the reconciliation of my velocity with tasks to iteration planning in Rally.

Anyway, this little system works pretty well for me. While at a high level it mirrors Rally, I also get the benefits of sub-tasks and user story templates, as well as integration with non-Zenoss-related actions.

How do you reconcile your Agile project with personal task management? I'd love to hear your methods in the comments, OmniFocus or no.

OmniFocus + Evernote

2010-02-17T12:00:00.003-05:00

A standing opinion about Evernote is that it's fantastic at accepting information but terrible at giving it up again. You can search from within the app, but any kind of integration with other apps is only barely possible—by, say, linking to the Evernote web client.

OmniFocus, for its part, is exceptional at helping you organize your life, but really isn't designed to be a repository of detailed information, so integration with Evernote is a natural desire. For example, I have an OmniFocus single-action list which I use to store blog post ideas; it's a natural next step to have an Evernote note associated with each containing notes and drafts.

So here's the method (admittedly, somewhat manual) that I've been using to keep references to Evernote notes with associated OmniFocus actions.

Create the note. This is the part that really ought to be automated, but isn't (some AppleScript-fu, which I unfortunately do not possess, could probably eliminate or ease this step. Suggestions welcome). Go into Evernote and make a new note, presumably titled similarly to your OmniFocus action.
Pull up the note in Finder. Herein lies the only interesting part. Since Evernote notes show up in Spotlight searches, there is at least some avenue for getting a reference to the note outside of Evernote itself; we will exploit this.
- Open up Finder and start a new search (type a character in the search box, then delete it).
- Click the + button to add a new criterion, where Kind is Other. Type "Evernote Note" into the box. You can restrict this search to your home folder if you have other Evernote users on the same computer. Sort descending by Last Opened.
- Click "Save." Save it as "Evernote" and add it to your sidebar.
Now selecting this saved search in Finder will pull up all your notes in an order conducive to this process. If you just created the note, it'll be right on top; otherwise, you can progressively filter the search by some text.
Drag the note to the OmniFocus. Just drop it on the action to get a link to Evernote in the notes field.

Nothing fancy, nothing new, and as I said, still more manual than one would hope. A button in OF that automatically creates a note in Evernote and plugs the reference into the action would be sweet. Still, until somebody writes that or I manage to find the time to learn some AppleScript, this method seems to be working pretty well for me.

Zenoss daemon status in bash prompt

2010-02-10T11:29:00.001-05:00

Seeing as how I bring Zenoss processes up and down forty times a day, I can get confused about what's running, and zenoss status gives me a bunch of stuff I don't care about. So I wrote this bash function to output a simple string representing my running daemons.

zends () {
    RESULT=""
    OUTPUT="`ps aux | grep $ZENHOME`"
    SEARCHFOR="zeoctl:z
               (zopectl|Startup/run.py):Z
               zenhub:h
               zenjobs:j"
    for f in $SEARCHFOR; do
        if [ -n "`echo \"$OUTPUT\" | egrep \"${f%:*}\"`" ]; then
            RESULT=${RESULT}${f#*:}
        fi
    done
    echo $RESULT
}

This iterates over regex:letter pairs and checks the ps output to see if a process matches. If so, it appends the letter to the string.

Then I added it into my prompt in ~/.bashrc:

PS1="[...snip...]$(zends)[...snip...] $"

Now my prompt looks like this:

ian | zhj | 10.42.1.7 | core/Products (devdetail) $

And I can tell at a glance that ZEO, zenhub and zenjobs are running, but not Zope.

extdirect 0.4 released: Batched requests, new licensing

2010-02-06T20:38:00.000-05:00

Sorry it's taken so long, those of you who've made requests and opened tickets, but finally, there's an update to extdirect. This release adds support for batched requests as described in the Ext.Direct spec (multiple calls within a certain amount of time are gathered into a single request).

In addition, I've removed the actual ExtJS code itself so this could be released under a less restrictive license. Ext 3.x is now merely a prerequisite.

Thanks to Brian Edwards and Jon-Pierre Gentil for their work on the analogous code in Zenoss, a version of which made its way into this release.

Run pyflakes/jslint automatically in Vim

2010-02-03T10:05:00.002-05:00

As maybe you can tell from the infrequency of updates to this blog, I (and the others with whom I'm working) have been churning out a huge amount of code recently, split roughly evenly between Python and JavaScript. One nice thing I got used to during the month I tried to switch from Vim to TextMate was a plugin that ran pyflakes every time I saved a Python file. After I gave up and went back to Vim, I missed that check. Luckily, other, more vimscript-savvy coders had the same idea. Here's how to set up both pyflakes and jslint to run on save.

pyflakes. I installed this excellent plugin:
```
$ git clone git://github.com/kevinw/pyflakes-vim.git
$ mkdir -p ~/.vim/ftplugin/python
$ cp pyflakes-vim/pyflakes.vim ~/.vim/ftplugin/python
```
I also followed his advice and installed his fork of pyflakes, which is incredibly faster than the original version:
```
$ git clone git://github.com/kevinw/pyflakes.git
$ cd pyflakes
$ sudo python setup.py install
```
This plugin actually highlights errors inline as you edit, and puts details in the status bar when you're over a line with an error:
jslint. This plugin is easier to install, though it doesn't have the same on-the-fly checking as the pyflakes plugin.
```
$ git clone git://github.com/hallettj/jslint.vim.git
$ cd jslint.vim
$ rake install
```
Then you can run jslint easily from within Vim:
```
:JSLint
```
That'll give you a nice quickfix window with all your extant problems.

To have it run on save, simply add to ~/.vimrc:
```
autocmd BufWritePost *.js JSLint
```
Finally, you can also pass options into jslint by putting them in ~/.jslintrc. For example, because I'm working with a few global namespaces/functions defined in other files, I have:
```
/* global Ext, Zenoss, _t */
```
That way, you can eliminate any noise and get right to the trailing commas.

I may try my hand at hacking the jslint plugin to do the same inline highlighting as the pyflakes plugin.

Anyway, there it is. Definitely helps keep the silly typos down.

Ext.Direct remoting in Django

2009-09-20T22:53:00.000-04:00

Well, as I didn't have a lot of other things weighing on me today, I created a Django implementation of the extdirect package I released earlier today. It's slightly more involved to set up, probably due to my not knowing Django nearly as well as Zope (suggestions welcome, Django enthusiasts), but after the initial setup writing the router classes is just as easy. Here's how to do it.

Add 'extdirect.django' to INSTALLED_APPS in settings.py. This of course assumes you have extdirect installed.
In views.py, define your router class and register it:
```
from extdirect.django import DirectRouter, register_router

class MyRouter(DirectRouter):
    def uppercase(self, word):
        return word.upper()
    def lowercase(self, word):
        return word.lower()

register_router(MyRouter, 'Remote')
```
The arguments to register_router are the router class, the client-side
namespace, and an optional url under /extdirect at which the router should be
available (defaults to the name of the class).
In the __init__.py of your app, add:
```
import views
```
This is so that the router classes will be registered on startup; it should
have no other effect.

In the root URLconf, map the extdirect urls by adding:

(r'^extdirect/', include('extdirect.django.urls'))

In your template, load the provider definitions:
```
{% load direct_providers %}
    {% direct_providers %}
```
If you don't have Ext on the page already, you can write a stripped-down
version directly to the page by adding +direct.js to the template tag:
```
{% direct_providers +direct.js %}
```
That's it. You should now have access on that template to the remote
methods:
```
Remote.MyRouter.uppercase({word:'a word'}, callback); 
```
```
  
```

After that, as long as you put your router classes in views.py and register them with register_router, everything should be available in the client automatically. Neat, no?

If any Djangoers out there know how to do this more elegantly (in particular, some way to obviate the need to import views in __init__.py would be nice) please let me know. For now, though, I think it's pretty useful.

Ext.Direct remoting in Zope

2009-09-20T12:47:00.003-04:00

With ExtJS 3.0 came Ext.Direct, an excellent library for remoting server-side methods to the client side. We had already made the decision to switch to Ext with the revamped Zenoss UI, so I'd been working with Ext quite a lot; in the upcoming 2.5 release, there's a brand-new event console that makes heavy use of Ext.Direct. After learning its ins and outs with all that work, I decided to write a Python version of the server-side component to make it easier to use; I then went further and created a Zope 2 and 3 compatible component that makes it trivial. extdirect is the result. Here's some brief instruction on how to use it.

In a nutshell, Ext.Direct allows one to create a client-side API that maps to a server-side API. One creates a "router" on the server, available at a given URL, that can accept requests specifying a method name, route them to the correct method, and return the result. As a brief, non-working example (ignoring framework), if the server-side router provided some methods:

class Router(object):
    def uppercase(self, word):
        return word.upper()
    def lowercase(self, word):
        return word.lower()

And that class were available at some URL, Ext.Direct lets you define a client-side provider that allows you to call those methods, without mapping each method to a separate URL:

function callback(result) {
    ... do something ...
}
Remote.Router.uppercase({word:'ext.direct'}, callback);
Remote.Router.lowercase({word:'EXT.DIRECT'}, callback);

Ideally, the server would be able to create the provider definition by inspecting the class, generating JavaScript that could be inserted into a template. extdirect provides a router base class and a function that can generate its provider definition.

I'll focus on the Zope-specific implementation, because it makes it so easy, but the base class can be used in any framework. I'm thinking I'll add a Django implementation next.

So, here's how to hook up a router in Zope, assuming extdirect is installed and available:

Define your class. DirectRouter is a subclass of BrowserView, so use self.context and self.request at will. Let's say this goes into utils.py:

from extdirect.zope import DirectRouter

class Utils(DirectRouter):
    
    def contextid(self):
        return self.context.id

    def uppercase(self, word):
        return word.upper()

Make sure to return something JSON-serializable.

Register the class as an Ext.Direct router in ZCML. This step both registers a view for the router and registers a viewlet containing the JavaScript required to create the client-side provider (against the included extdirect viewlet manager):
```
<configure xmlns="http://namespaces.zope.org/browser">

     <include package="extdirect.zope" file="meta.zcml"/>

       <directRouter 
           class=".utils.Utils" 
           name="remote_utils" 
           namespace="Remote"
           />

</configure>
```
The class, of course, refers to the DirectRouter subclass you defined in step 1. The name is like a normal BrowserView name, and will be the URL at which your router will be available. The namespace is the namespace in the client at which the methods will be accessible, and can be longer (if you already have a namespace set up, you could do "Zenoss.server.remote" or whatever you like). This provider will be available at [namespace].[class name], so in this case, Remote.Utils.
Include the viewlet manager in a page template. You have two options here. If you've already got Ext loaded through other means (say it's the framework you're using on the site), you can use:
```
<tal:block tal:content="structure provider:extdirect"/>
```
If you're not using Ext already, and don't want to, you can use a stripped-down version including only those components necessary for Ext.Direct, which is included in extdirect and registered as a resource and a viewlet:
```
<tal:block tal:content="structure provider:extdirect+direct.js"/>
```

Call methods at will. That's all. You can now call methods on that namespace and class:

Remote.Utils.contextid({}, callback);
Remote.Utils.uppercase({word:'A word'}, callback);

And that's all there is to it. It couldn't be simpler. Your router is available at ./remote_utils, but Ext.Direct handles all the communication so you don't ever really need to use it yourself. Further details of the base class are available at the project page on PyPI. Code is hosted on Google Code.

Vim, Part II: Matching Pairs

2009-07-06T10:00:00.007-04:00

One of the more standard vim tweaks is to have parens, braces, brackets, and quotes auto-close—that is, you type (, and the ) is inserted automatically. This can be a big help in keeping track of nested groups. This is easy to do poorly; to do it well takes a little more thought. I took a look at TextMate's rules and mimicked them (mostly). Put all this stuff in your .vimrc (_vimrc on Windows).

Note: A couple of these tweaks are modified or copied outright from the Vim Scripts repository.

Auto-close pairs

The initial auto-close is quite easy; leaving the group is slightly more involved. If you're inside parens, at the end, and you want to leave them, instinct is just to type the closing ); without a little logic, however, that will in no way achieve the desired result. The rule is: if typing the closing character would end the block without auto-close (that is, because it's the character to the immediate right of the cursor), it should end it with auto-close. Here's how you achieve that.

Map the auto-close for non-quotes. This is trivial:
```
inoremap ( ()<Left>
inoremap [ []<Left>
inoremap { {}<Left>
autocmd Syntax html,vim inoremap < <lt>><Left>
```
No big deal; when the opening character is typed, insert the closing character and hit <Left> so the cursor is between them.

Function to check next character.

function! ClosePair(char)
if getline('.')[col('.') - 1] == a:char
return "\<Right>"
else
return a:char
endif
endf

Map closing characters. Now, all you need to do is link the closing characters to the function:

inoremap ) <c-r>=ClosePair(')')<CR>
inoremap ] <c-r>=ClosePair(']')<CR>
inoremap } <c-r>=ClosePair('}')<CR>

Map the auto-close for quotes. This is only slightly more difficult than ()[]{}, because of the case where you might add an escaped quotation mark to a string. It requires a separate function: since the opening and closing characters are the same, we can't map the closing to ClosePair. So we put it all in one function:

function! QuoteDelim(char)
let line = getline('.')
let col = col('.')
if line[col - 2] == "\\"
"Inserting a quoted quotation mark into the string
return a:char
elseif line[col - 1] == a:char
"Escaping out of the string
return "\<Right>"
else
"Starting a string
return a:char.a:char."\<Left>"
endif
endf

Then just need to map that:

inoremap " <c-r>=QuoteDelim('"')<CR>
inoremap ' <c-r>=QuoteDelim("'")<CR>

Dump all that in to .vimrc, source it or restart vim, and try typing some stuff! You may notice a few places where it isn't quite what you'd want, though, which brings us to our next two tweaks.

Wrap visually selected text

One of the nicer minor features of TextMate is its treatment of highlighted text. If you have something highlighted and type a, it replaces the text, like other editors. If you type (, however, it wraps the selected text in parentheses. This is enormously useful. Luckily, it's very easy to recreate in Vim:

vnoremap (  <ESC>`>a)<ESC>`<i(<ESC>
vnoremap )  <ESC>`>a)<ESC>`<i(<ESC>
vnoremap {  <ESC>`>a}<ESC>`<i{<ESC>
vnoremap }  <ESC>`>a}<ESC>`<i{<ESC>
vnoremap "  <ESC>`>a"<ESC>`<i"<ESC>
vnoremap '  <ESC>`>a'<ESC>`<i'<ESC>
vnoremap `  <ESC>`>a`<ESC>`<i`<ESC>
vnoremap [  <ESC>`>a]<ESC>`<i[<ESC>
vnoremap ]  <ESC>`>a]<ESC>`<i[<ESC>

Backspace in empty pair deletes both

Now that you have auto-closing parens, let's say you type one by mistake. Unfortunately, deleting it now involves two extra keystrokes (<ESC>xx or <Right><Bksp><Bksp> instead of <Bksp>). TextMate gets around this by deleting both if you're in an empty block. I was tired of mapping the same thing for every pair, though, so I wrote a function that uses the matchpairs option, which exists to tell Vim what delineates a discrete block, to figure it out dynamically (thanks to Aristotle Pagaltzis for cleaning it up!). It checks whether the characters on either side of the cursor constitute a valid pair:

function! InAnEmptyPair()
let cur = strpart(getline('.'),getpos('.')[2]-2,2)
for pair in (split(&matchpairs,',') + ['":"',"':'"])
if cur == join(split(pair,':'),'')
return 1
endif
endfor
return 0
endfunc

And then the actual deletion function:

func! DeleteEmptyPairs()
if InAnEmptyPair()
return "\<Left>\<Del>\<Del>"
else
return "\<BS>"
endif
endfunc

Then map it to Backspace, so it's called whenever Backspace is:

inoremap <expr> <BS> DeleteEmptyPairs()

So, in plain English: when Backspace is hit, we check to see if the characters on the immediate left and right of the cursor are part of a matching pair, looking at the matchpairs option to see what counts of a valid pair. If we are in a pair, delete both characters; otherwise, send a normal Backspace.

Next (and probably last) in the series: Whatever's left in my .vimrc, namely, assorted tweaks to make one's coding life easier.

Vim, Part I: Improved Python syntax, Blackboard color scheme

2009-07-03T08:02:00.011-04:00

Every few months, I notice that TextMate exists and try to use it; inevitably, the requirement that I use arrow keys defeats the endeavor. It has a few features, however, that I like, most of which I've ported to vim, my primary text editor. These features are too numerous for a single post, so I'll highlight them in a series. First up: An improved Python syntax definition and my favorite color scheme.

Python Syntax

The most popular Python syntax highlighting script is pretty complete, but it's missing a few things (Update: several of the improvements I made a year ago have been added to Dmitry Vasiliev's script, such as decorators, doctests, imports, etc. I've merged the two, so this is slightly less special than before). Specifically: docstrings, self, arguments/base classes, assignment as distinct from other operators, and a few other little things. TextMate had these, so I ported them, and the results were pretty good.

Download the improved syntax file and drop it into ~/.vim/syntax.

Blackboard Color Scheme

TextMate had a color scheme that I loved: Blackboard. The colors were primaryish without being irritatingly bright or clashing; keywords, builtins, and class definitions were given appropriate prominence. I ported it to Vim. Here's what it looks like:

That example doesn't include a doctest, but they are highlighted separately in the same color as builtins.

Download the color scheme and drop it into ~/.vim/colors, then do :colorscheme blackboard to see the results. Pretty easy on the eyes, no?

Next in my Vim series: automatic handling of pair completion (and deletion).

Plaintext task list from OmniFocus with AppleScript

2009-06-06T11:32:00.009-04:00

I live in OmniFocus most of the day. I have lots of perspectives set up to help me parse my various responsibilities (see my previous post, "How I Use OmniFocus," for more details); in particular, I have one perspective called "Global Tasks" (View Mode: Context, Filter: Active, Grouping: Ungrouped, Sorting: Due, Action Filter: Next Action) that essentially gives me a list, ordered by dueness, of what I should be doing right now.

It helps me focus to have that list of tasks constantly in front of my eyes. This is counterbalanced by my desire to save screen real estate for the assorted terminal windows and browsers I need up to do my job; OmniFocus takes up valuable real estate. So I set about writing a quick script to pull tasks from OmniFocus, format them, and return them as plain text, which I could embed on my desktop with GeekTool.

set taskList to ""
tell application "OmniFocus"
    tell the default document to tell the front document window
        set perspective name to "Global Tasks"
        set oTrees to trees of content
        set n to count of oTrees
        repeat with i from 1 to n
            set oTask to value of (item i of oTrees)
            set taskTitle to name of oTask
            set projTitle to name of containing project of oTask
            set taskList to taskList & taskTitle & 
                "  (" & projTitle & ")\n"
        end repeat
    end tell
end tell
return taskList

I saved that as ~/src/omnifocus_tasks.scpt and added a Command item to GeekTool that ran osascript /Users/ian/src/omnifocus_tasks.scpt every minute. The result was exactly what I wanted:

Now, even if I hide OmniFocus to make room for something else, I can't forget to check for my next tasks and keep everything up to date.

Rapid development egg to RPM with buildout

2009-06-01T09:00:00.002-04:00

I'm developing a feature for Zenoss that lives in its own Zope product. For the time being, it's fairly independent of Zenoss and can be demoed in a separate Zope instance. So I have a buildout for the Zope environment, also setting up a copy of the new product as a development egg (see this tutorial for an example of how to set that up). I have that development egg pulled into the Zope demo buildout via svn:externals.

At certain points in the development cycle, however, I want to send QA an RPM of the demo instance, with my package installed, so they can test it. This is kind of a hassle if I have to build the egg of the product and get it in the buildout directory on the build box in order to generate an RPM. Nor can I simply leave it as a development egg, because the package wouldn't be in the same place on the test box, so the link would fail.

So here's an easy way to turn a development egg into a production egg for the purposes of creating an RPM, using a separate buildout config file.

You'll want to read this tutorial on basic RPM creation using zc.buildout. In addition, the buildout.cfg will include a develop option in the buildout section:

[buildout]
...
develop = src/Products.MyCustomProduct

(src exists in my demo directory, and has the svn:externals definition pointing at the custom package.) The egg is also included in the instance section:

[instance]
...
eggs = 
    ${buildout:eggs}
    Products.MyCustomProduct

zcml = Products.MyCustomProduct

This gets me a development buildout, in which I can work on the package in src/Products.MyCustomProduct without building eggs or anything.

Now when it comes time to build the RPM, we need to modify the rpm config file from the tutorial to build Products.MyCustomProduct as an egg, then install it. We'll use the z3c.recipe.egg, which lets you run setup.py in a source package with arbitrary arguments. Here's the new rpm.cfg:

[buildout]
extends = buildout.cfg
develop = 
parts = 
    MyCustomProduct-build
    zope2
    instance
    zopepy
    myzope

[MyCustomProduct-build]
recipe = z3c.recipe.egg:setup
setup = src/Products.MyCustomProduct
args = bdist_egg

[instance]
find-links = 
    ${buildout:find-links}
    ${MyCustomProduct-build:setup}/dist

[myzope]
...

Explanation. So, the egg is already referenced in buildout.cfg as one of the instance eggs. All we do in this one is remove its declaration as a development egg (with develop= in the buildout section), add a part to create the egg artifact (MyCustomProduct-build), and tell the instance section where to find it (expanded find-links).

z3c.recipe.egg just needs the setup option defined, which points at a package directory containing setup.py. It'll install by default, whereas we just want it to build an egg, so we specify custom args with the args option.

Notice that we have to build the egg first. This is obvious upon reflection, but the first time I just added MyCustomProduct-build to the parts instead of making sure it was first, which meant that buildout couldn't find it in the instance step, because it didn't exist yet.

This will create a deployment instance just fine on its own, in the current directory. But in the RPM example, we build out somewhere else, so we'll have to update the buildout command in the spec file so that it can tell buildout where to find the source of the custom package. This is a modification to the %build script:

%build
...
bin/buildout -c rpm.cfg buildout:directory=%{installdir} MyCustomProduct-build:setup=%{sourcedir}/src/Products/ZenUI3

Then we build the RPM as in the tutorial. This time, it'll package up the egg in its current state and install that artifact into the Zope instance. This means that I can generate an RPM for QA on the fly with a single command, which is lots easier than assembling the parts manually.

Using zc.buildout to generate an RPM

2009-05-31T16:00:00.005-04:00

zc.buildout is a project that allows one to define the steps to create an application environment in config files. It is, roughly, a Makefile replacement, or at least can be used that way. It makes it very easy to reproduce deployments, both for development and for production, with nothing but a bootstrap and a config file.

The natural next step, of course, would be to use buildout to generate installable packages like RPMs. This is less straightforward than one might expect. There are probably many ways to do it, as buildout is very versatile; I'll explain one method I've come up with after the jump.

(My rpm knowledge is pretty minimal, so this may not be strictly according to doctrine, but it seems to work flawlessly in the scenarios I've tested.)

I'll use as an example the installation of a vanilla Zope 2 instance, with an rc script to control it. One may of course add one's own products to the buildout, thereby creating a custom application; that is beyond the scope of this example (see Martin Aspeli's excellent Plone-oriented but generally informative tutorial, along with the buildout docs, etc.).

Here's the config for a Zope 2.11.2 buildout:

[buildout]
parts =
    zope2
    instance
    zopepy

find-links =
    http://dist.plone.org
    http://download.zope.org/distribution/
    http://effbot.org/downloads

# Add additional eggs here
eggs =

# Reference any eggs you are developing here, one per line
# e.g.: develop = src/my.package
develop =

[zope2]
recipe = plone.recipe.zope2install
url = http://www.zope.org/Products/Zope/2.11.2/Zope-2.11.2-final.tgz

[instance]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
user = admin:admin
http-address = 8080
debug-mode = on
#verbose-security = on

# If you want Zope to know about any additional eggs, list them here.
# This should include any development eggs you listed in develop-eggs above,
# e.g. eggs = ${buildout:eggs} ${plone:eggs} my.package
eggs =
    ${buildout:eggs}

# If you want to register ZCML slugs for any packages, list them here.
# e.g. zcml = my.package my.other.package
zcml =

products =
    ${buildout:directory}/products

[zopepy]
recipe = zc.recipe.egg
eggs = ${instance:eggs}
interpreter = zopepy
extra-paths = ${zope2:location}/lib/python
scripts = zopepy

Notice I'm using Plone's recipes for setting up a Zope instance; they work as well as you might hope. If you bootstrap and buildout from this config file, you'll get yourself a Zope server listening at 8080, and a custom interpreter with the right paths and everything.

Make a new directory containing the above config file, saved as buildout.cfg, and a copy of bootstrap.py. If you like, you can make sure it works with:


$ python bootstrap.py
$ bin/buildout
$ bin/instance fg

That'll start up Zope in the foreground.

A very nice feature of zc.buildout is the ability of config files to extend others. We'll use this to create a separate buildout with the extra/different features we need for an RPM. In this minimal example, the differences are few, but you can see how it would work well for more complex buildouts. Save this as rpm.cfg in the same directory as buildout.cfg:


[buildout]
extends = buildout.cfg
parts += myzope

[myzope]
recipe = zc.recipe.rhrc
parts = zoperc

[zoperc]
run-script = ${buildout:directory}/bin/instance

Building out with this would do the same thing as our buildout.cfg, but would also install /etc/init.d/myzope as a control script, thanks to zc.recipe.rhrc.

Now we get to the (slightly) tricky part. Normally, one would build an RPM by creating a source tarball, dropping it in /usr/src/redhat/SOURCES, and pointing a spec file at it. We could do that here if we wanted to. We could also use zc.sourcerelease to create a tarball, including egg dependencies and whatnot, and do the same thing (though that irritatingly requires one to build out twice). But it's an extra step.

The directory option to buildout lets you build out into a directory other than the current. You can also override the config file from the command line. So the simplest method to create an RPM is just to pass the install path to buildout in the spec file. Here's a spec file that works (save it as myzope.spec in the same directory as rpm.cfg):

%define name myzope
%define sourcedir %(echo $PWD)
%define installdir /opt/%{name}
%define rcscript /etc/init.d/%{name}

Name: %{name}
Version: 1.0
Release: 0
Summary: My Zope Instance
URL: http://www.zope.org
License: GPL
Vendor: Me
Packager: Me <me@example.com>
Group: Applications/Database
Buildroot: /tmp/%{name}-buildroot

%description
%{summary}

%prep
rm -rf %{rcscript} $RPM_BUILD_ROOT %{installdir}
mkdir -p $RPM_BUILD_ROOT %{installdir}

%build
cd %{sourcedir}
python bootstrap.py
bin/buildout -c rpm.cfg buildout:directory=%{installdir}

%install
echo "effective-user nobody" >> %{installdir}/parts/instance/etc/zope.conf
mkdir %{installdir}/products
mkdir -p $RPM_BUILD_ROOT/etc/init.d
mkdir -p $RPM_BUILD_ROOT%{installdir}
mv %{rcscript} $RPM_BUILD_ROOT%{rcscript}
mv %{installdir}/* $RPM_BUILD_ROOT%{installdir}

%files
%defattr(-, root, root, 0755)
%{rcscript}
%defattr(-, nobody, nobody, 0755)
%{installdir}

%clean
rm -rf %{rcscript} $RPM_BUILD_ROOT %{installdir}

Now, step by step. First of all, the clever bit:


%define sourcedir %(echo $PWD)

This lets us find the directory with the buildout config—provided, of course, that the spec file lives in the same directory as rpm.cfg, and that you run rpmbuild from there. Notice that we also define the directory in which this will be installed, namely /opt/myzope.

The rest is basic spec file stuff, until you get to the build script:

%build
cd %{sourcedir}
python bootstrap.py
bin/buildout -c rpm.cfg buildout:directory=%{installdir}

So we're using the rpm config file, but overriding the directory option in the buildout section to use /opt/myzope as the location in which to build out the environment. We have to do it this way—instead of, say, building out in $RPM_BUILD_ROOT—because the paths in the buildout would then refer to /tmp/myzope-buildroot/opt/myzope instead of /opt/myzope.

Then it's as simple as doing a couple Zope-specific steps (setting the user who runs Zope), copying the resulting environment into the build root (not forgetting /etc/init.d/myzope) and letting it be packaged up.

Build it with rpmbuild -bb myzope.spec. Shizam, an RPM that you can install. Start it up with /sbin/service myzope start and check it out in the browser.

There are a few other tricks involved in separating a more complex development buildout from one that is ready for an RPM. I'll explain further in my next post.

cliutils has a user not named Ian!

2009-04-23T11:53:00.001-04:00

Jason Friedland has very kindly submitted my cliutils package for inclusion in Fedora. Thanks! Pretty exciting to learn that somebody is using it. Now I'm trying to come up with ways to improve it. Any ideas for features to add, tens of readers?

Human-comparison sort in Python

2009-03-10T11:00:00.002-04:00

I recently had a need to sort a large list of items (I admit freely that this list comprised episodes of Battlestar Galactica) by a subjective criterion (my liking). This turned into a minor interface problem; how best to sort these without a) a Netflix queue interface or b) dragging Excel rows around? Also, how could I implement an efficient sorting algorithm, rather than waste time with the simplest bubble sort?

The solution I came up with was almost too simple to feel proud of. Python uses an adaptive mergesort by default, which was plenty good enough for my purposes. All I had to do, then, was have it ask a human to compare each pair of elements.


QUESTION_TEMPLATE = """
1. %s or
2. %s or
3. neither? [neither] """

def human_cmp(a, b):
    answer = raw_input(
        QUESTION_TEMPLATE % (a, b))
    if answer=="1":   return -1
    elif answer=="2": return 1
    else:             return 0

EPISODES = open('bsg_episodes.txt').readlines()
EPISODES.sort(human_cmp)
for i, title in EPISODES:
    print '%s. %s' % (i, title)

So it just uses the built-in algorithm and asks you for each comparison. Let me tell you, way better than creating a single-elimination tournament bracket in Excel.

Upgrade bash to 4.0 in Mac OS X

2009-03-09T07:00:00.000-04:00

bash 4.0 was released last month, and I'm only now getting around to messing with it. So far the things I use most are the '**' recursive globbing and the 'autocd' option. Here's how I upgraded my system and a few ways I use the new features.

Download, build and install. Get the source tarball:
```
$ wget ftp://ftp.cwru.edu/pub/bash/bash-4.0.tar.gz
```
Build and install:
```
$ tar xzf bash-4.0.tar.gz
$ cd bash-4.0
$ ./configure && make && sudo make install
```
On my system (Leopard + Xcode) I had no trouble compiling. The binary is installed at /usr/local/bin/bash. You can also install via MacPorts or Fink, I imagine, but I try to do these things myself when possible, if only for the sake of transparency.

Configure to use the new shell. First, register the new binary as a valid shell:
```
$ sudo bash -c "echo /usr/local/bin/bash >> /private/etc/shells"
```
Then change your user to use it as its shell with chsh (this is blatantly obvious, but modify the "Shell" line to point to /usr/local/bin/bash).
Now open a new shell (restarting Terminal.app will do, or just Cmd-N for a new window) and make sure your changes took:
```
[ian@iansmbp] ~/> echo $SHELL
/usr/local/bin/bash
[ian@iansmbp] ~/> echo $BASH_VERSION
4.0.0(1)-release
```

Enable the new features. There's a bunch of features in the 4.0 release that I haven't gone through yet, including coproc, improved programmable completion, a new &>> redirect operator (synonym of the old >>myfile 2>&1 pattern) and new case-insensitive expansion options. You can peruse a list of the new stuff at your leisure. For immediate gratification, however, just turn on recursive globbing and autocd:
```
$ echo "shopt -s globstar autocd" >> ~/.profile
$ source ~/.profile
```
** matches contents recursively. For example, where you might previously have recursively removed all your byte-compiled Python modules with:
```
$ find . -name \*.pyc | xargs rm -f
```
You can now simply:
```
$ rm -f **/*.pyc
```
If you've got a deep directory structure, and you want to get into a subdirectory whose name you know, you can replace:
```
$ cd path/to/the/directory/i/want/named/mydir
```
with:
```
$ cd **/mydir
```
Hey, and if those extra three characters at the beginning of that last command are too much for you, then you'll love the autocd option, which, when enabled, permits you to cd to a directory merely by typing its name:
```
[ian@iansmbp] ~/src/zenoss/core/Products/> **/yui
cd ZenWidgets/skins/zenui/yui
[ian@iansmbp] ~/src/zenoss/core/Products/ZenWidgets/skins/zenui/yui/>
```

More on new features as warranted; these two, however, are those of zsh that I missed most.

How Joseph Method Stole My Day

2009-03-07T14:26:00.007-05:00

My friend Method shared this with me via Google Reader this morning, accompanied by the provocative message, "Here you go, Ian." The problem, in short, was to discover the first ten-digit prime in consecutive digits of e. And a day of planned work-catching-up dissolved before it began.

I had a solution in twenty minutes or so, using third-party libraries pycrypto (for isPrime) and mpmath (for e), but that made the whole enterprise trivial and didn't quite seem cricket. Also, it was slightly inefficient, because, being limited by the code of others, I had to calculate e to n digits first, and then check that for primes; I figured if I could calculate digits of e lazily I'd get an extremely minor but nonetheless pride-inducing speed increase.

To Wikipedia! where I learned scads about primality tests (luckily I didn't read the page on the constant e until later, or I would have learned that this was a common puzzle that pretty much everybody tries to solve. Also I would have learned the answer). I remembered how to calculate e using the Taylor series 1 + 1/1! + 1/2! + 1/3!..., so that wasn't a problem—until I ran into precision woes.

That led me down a dark path whereon I attempted to program arbitrary-precision arithmetic from scratch, and learned all about mantissas and bitwise operators over the course of five hours. I got pretty far with that before I realized that I was spending my entire day trying to comprehend what I imagine amounts to at least a month of college-level CS. I went to school for other things, like French poetry.

Also at this point I remembered the existence of Python's own decimal floating-point arithmetic package, decimal, which had apparently dropped completely out of my brain exactly when I needed it. Thankfully, it resurfaced before I had to give up on this psychotic enterprise altogether.

After that it was easy. I post the code below not because I believe it is in any way notable—I'm sure it's old hat to those who went to school for this—but because I just want someone to know that I actually did something today. Thanks, Method.


from random import randint
from operator import mul
from collections import deque
from decimal import Decimal, getcontext

# Allow enough precision to get an answer 
getcontext().prec = 1000

def factorial(x): 
    if x==1: return x
    return reduce(mul, xrange(2, x+1))

def taylor_series():
    """
    e == 1/1! + 1/2! + 1/3! ...
    """
    s = Decimal(1)
    _e = Decimal(1)
    while True:
        _e += (Decimal(1)/factorial(s))
        yield _e 
        s += 1

def digits_of_e():
    """
    Yield digits of e lazily by noticing when
    they stop changing as the Taylor series 
    converges.
    """
    idx = 2 # skip '2.'
    last = None
    for iteration in taylor_series():
        i = str(iteration)
        try:
            cur = i[idx]
        except IndexError:
            continue
        if cur==last:
            idx += 1
            yield cur
        last = i[idx]

def ten_digit_chunks():
    """
    Collect digits of e as they are discovered
    and yield them in ten-digit chunks.
    """
    l = deque()
    for digit in digits_of_e():
        l.append(digit)
        # Keep the stack at 10 by popping
        # from the left
        while len(l)>10:
            l.popleft()
        if len(l)==10: 
            yield ''.join(l)

def is_prime(n, k=20):
    """
    Simple implementation of the Miller-Rabin 
    primality test. Thanks, Wikipedia!

    n is the integer to be tested; k is the 
    precision (number of random possible 
    witnesses to test against)
    """
    # Find s and d such that (2**s)*d==n
    d, s = n-1, 0
    while not d & 1: # d & 1 == d % 2, only faster
        d >>= 1 # d >> 1 == d / 2, only faster
        s += 1
    # Test for k random numbers
    for i in xrange(k):
        _c = False
        a = randint(2, n-2)
        x = pow(a, d, n) # a**d % n
        if x==1 or x==n-1:
            continue
        for r in xrange(1, s):
            x = pow(x, 2, n) # x**2 % n
            if x==1:
                return False
            elif x==n-1:
                _c = True
                break
        if _c: continue
        return False
    return True

def main():
    for chunk in ten_digit_chunks():
        print "Trying %s..." % chunk
        if is_prime(long(chunk)):
            print "="*20
            print
            print "Found! First ten-digit"
            print "prime in consecutive digits"
            print "of e is %s!" % chunk
            print
            print "="*20
            break

if __name__=="__main__":
    main()

UPDATE: In order to get through this ridiculous set of problems to the unsatisfying and insulting end, you can use this implementation of the Brent-Salamin algorithm to find primes in digits of pi, replacing taylor_series in the digits_of_e function in the code above:


def brent_salamin():
    """
    A series whose sum converges on pi
    incredibly quickly.
    """
    a = Decimal("1.")
    b = Decimal("1.")/(Decimal("2.").sqrt())
    t = Decimal("1.")/Decimal("4.")
    p = Decimal("1.")
    while True:
        a1 = Decimal(a + b)/Decimal("2.")
        b1 = Decimal(a*b).sqrt()
        t1 = t - p*((a - a1)**2)
        p1 = 2*p
        yield ((a1+b1)**2)/(t1*Decimal("4."))
        a, b, t, p = a1, b1, t1, p1