PythonLovers.com

People who don't have 100% code coverage are idiots

Sun, 22 Feb 2009 23:17:37 GMT

My last post, Good code coverage: Necessity vs Sufficiency, about how you should maintain high code coverage with your automated tests, seems to have really struck a nerve in a small group of people -- I got some fantastic comments, with some great pointers. Michael Foord's comment, 'Too often "testing is no silver bullet" is used as an excuse not to test' hit the nail on the head, and Andrew Dalke linked to his excellent & detailed discussion with Glyph about code coverage, as well as pointing out the SQLite testing policy which is, frankly, astounding.

But I'm not writing this just to point out those links. No, this is partly in response to Ned Batchelder's plaintive question about why I picked on him. I picked on him because of his article's title: "there are flaws in coverage measurement." Now, Ned is the author of coverage, the main code coverage tool available for Python -- and I know he thinks code coverage is really important. And it is entirely legitimate to write critical essays about code coverage, especially statement coverage. But I also get frustrated with people who write blog posts with even moderately sensational and misrepresentative blog post titles. And that's why I included him in the post complaining about such things.

Got that? I hate sensational and misrepresentative titles.

Good day.

--titus

PyMOTW: tarfile

Sun, 22 Feb 2009 22:44:57 GMT

tarfile – Tar archive access

Purpose:	Tar archive access.
Python Version:	2.3 and later

The tarfile module provides read and write access to UNIX tar archives, including compressed files. In addition to the POSIX standards, several GNU tar extensions are supported. Various UNIX special file types (hard and soft links, device nodes, etc.) are also handled.

Testing Tar Files

The is_tarfile() function returns a boolean indicating whether or not the
filename passed as an argument refers to a valid tar file.

import tarfile

for filename in [ 'README.txt', 'example.tar', 
                  'bad_example.tar', 'notthere.tar' ]:
    try:
        print '%20s  %s' % (filename, tarfile.is_tarfile(filename))
    except IOError, err:
        print '%20s  %s' % (filename, err)

Notice that if the file does not exist, is_tarfile() raises an IOError.

$ python tarfile_is_tarfile.py
          README.txt  False
         example.tar  True
     bad_example.tar  False
        notthere.tar  [Errno 2] No such file or directory: 'notthere.tar'

Reading Meta-data from an Archive

Use the TarFile class to work directly with a tar archive. It supports methods for reading data about existing archives as well as modifying the archives by adding additional files.

To read the names of the files in an existing archive, use getnames():

import tarfile

t = tarfile.open('example.tar', 'r')
print t.getnames()

The return value is a list of strings with the names of the archive contents:

$ python tarfile_getnames.py
['README.txt']

In addition to names, meta-data about the archive members is available as instances of TarInfo objects. Load the meta-data via getmembers() and getmember().

import tarfile
import time

t = tarfile.open('example.tar', 'r')
for member_info in t.getmembers():
    print member_info.name
    print '\tModified:\t', time.ctime(member_info.mtime)
    print '\tMode    :\t', oct(member_info.mode)
    print '\tType    :\t', member_info.type
    print '\tSize    :\t', member_info.size, 'bytes'
    print

$ python tarfile_getmembers.py
README.txt
        Modified:       Sun Feb 22 11:13:55 2009
        Mode    :       0644
        Type    :       0
        Size    :       75 bytes

If you know in advance the name of the archive member, you can retrieve its TarInfo object with getmember().

import tarfile
import time

t = tarfile.open('example.tar', 'r')
for filename in [ 'README.txt', 'notthere.txt' ]:
    try:
        info = t.getmember(filename)
    except KeyError:
        print 'ERROR: Did not find %s in tar archive' % filename
    else:
        print '%s is %d bytes' % (info.name, info.size)

If the archive member is not present, getmember() raises a KeyError.

$ python tarfile_getmember.py
README.txt is 75 bytes
ERROR: Did not find notthere.txt in tar archive

Extracting Files From an Archive

To access the data from an archive member within your program, use the extractfile() method, passing the member’s name.

import tarfile

t = tarfile.open('example.tar', 'r')
for filename in [ 'README.txt', 'notthere.txt' ]:
    try:
        f = t.extractfile(filename)
    except KeyError:
        print 'ERROR: Did not find %s in tar archive' % filename
    else:
        print filename, ':', f.read()

$ python tarfile_extractfile.py
README.txt : The examples for the tarfile module use this file and example.tar as data.

ERROR: Did not find notthere.txt in tar archive

If you just want to unpack the archive and write the files to the filesystem, use extract() or extractall() instead.

import tarfile
import os

os.mkdir('outdir')
t = tarfile.open('example.tar', 'r')
t.extract('README.txt', 'outdir')
print os.listdir('outdir')

$ python tarfile_extract.py
['README.txt']

Note

The standard library documentation includes a note stating that extractall() is safer than extract(), and it should be used in most cases.

import tarfile
import os

os.mkdir('outdir')
t = tarfile.open('example.tar', 'r')
t.extractall('outdir')
print os.listdir('outdir')

$ python tarfile_extractall.py
['README.txt']

If you only want to extract certain files from the archive, their names can be passed to extractall().

import tarfile
import os

os.mkdir('outdir')
t = tarfile.open('example.tar', 'r')
t.extractall('outdir', members=[t.getmember('README.txt')])
print os.listdir('outdir')

$ python tarfile_extractall_members.py
['README.txt']

Creating New Archives

To create a new archive, simply open the TarFile with a mode of 'w'. Any existing file is truncated and a new archive is started. To add files, use the add() method.

import tarfile

print 'creating archive'
out = tarfile.open('tarfile_add.tar', mode='w')
try:
    print 'adding README.txt'
    out.add('README.txt')
finally:
    print 'closing'
    out.close()

print
print 'Contents:'
t = tarfile.open('tarfile_add.tar', 'r')
for member_info in t.getmembers():
    print member_info.name

$ python tarfile_add.py
creating archive
adding README.txt
closing

Contents:
README.txt

Using Alternate Archive Member Names

It is possible to add a file to an archive using a name other than the original file name, by constructing a TarInfo object with an alternate arcname and passing it to addfile().

import tarfile

print 'creating archive'
out = tarfile.open('tarfile_addfile.tar', mode='w')
try:
    print 'adding README.txt as RENAMED.txt'
    info = out.gettarinfo('README.txt', arcname='RENAMED.txt')
    out.addfile(info)
finally:
    print 'closing'
    out.close()

print
print 'Contents:'
t = tarfile.open('tarfile_addfile.tar', 'r')
for member_info in t.getmembers():
    print member_info.name

The archive includes only the changed filename:

$ python tarfile_addfile.py
creating archive
adding README.txt as RENAMED.txt
closing

Contents:
RENAMED.txt

Writing Data from Sources Other Than Files

Sometimes you want to write data to an archive but the data is not in a file on the filesystem. Rather than writing the data to a file, then adding that file to the ZIP archive, you can use addfile() to add data from an open file-like handle.

import tarfile
from cStringIO import StringIO

data = 'This is the data to write to the archive.'

out = tarfile.open('tarfile_addfile_string.tar', mode='w')
try:
    info = tarfile.TarInfo('made_up_file.txt')
    info.size = len(data)
    out.addfile(info, StringIO(data))
finally:
    out.close()

print
print 'Contents:'
t = tarfile.open('tarfile_addfile_string.tar', 'r')
for member_info in t.getmembers():
    print member_info.name
    f = t.extractfile(member_info)
    print f.read()

By first constructing a TarInfo object ourselves, we can give the archive member any name we wish. After setting the size, we can write the data to the archive using addfile() and passing a StringIO buffer as a source of the data.

$ python tarfile_addfile_string.py

Contents:
made_up_file.txt
This is the data to write to the archive.

Appending to Archives

In addition to creating new archives, it is possible to append to an existing file. To open a file to append to it, use mode 'a'.

import tarfile

print 'creating archive'
out = tarfile.open('tarfile_append.tar', mode='w')
try:
    out.add('README.txt')
finally:
    out.close()

print 'contents:', [m.name 
                    for m in tarfile.open('tarfile_append.tar', 'r').getmembers()]

print 'adding index.rst'
out = tarfile.open('tarfile_append.tar', mode='a')
try:
    out.add('index.rst')
finally:
    out.close()

print 'contents:', [m.name 
                    for m in tarfile.open('tarfile_append.tar', 'r').getmembers()]

The resulting archive ends up with 2 members:

$ python tarfile_append.py
creating archive
contents: ['README.txt']
adding index.rst
contents: ['README.txt', 'index.rst']

Working with Compressed Archives

Besides regular tar archive files, the tarfile module can work with archives compressed via the gzip or bzip2 protocols. To open a compressed archive, modify the mode string passed to open() to include ":gz" or ":bz2", depending on the compression method you want to use.

import tarfile
import os

fmt = '%-30s %-10s'
print fmt % ('FILENAME', 'SIZE')
print fmt % ('README.txt', os.stat('README.txt').st_size)

for filename, write_mode in [
    ('tarfile_compression.tar', 'w'),
    ('tarfile_compression.tar.gz', 'w:gz'),
    ('tarfile_compression.tar.bz2', 'w:bz2'),
    ]:
    out = tarfile.open(filename, mode=write_mode)
    try:
        out.add('README.txt')
    finally:
        out.close()

    print fmt % (filename, os.stat(filename).st_size),
    print [m.name for m in tarfile.open(filename, 'r:*').getmembers()]

When opening an existing archive for reading, you can specify "r:*" to have tarfile determine the compression method to use automatically.

$ python tarfile_compression.py
FILENAME                       SIZE
README.txt                     75
tarfile_compression.tar        10240      ['README.txt']
tarfile_compression.tar.gz     213        ['README.txt']
tarfile_compression.tar.bz2    186        ['README.txt']

PyCon 2009: In ur brain, giving you the pythons

Sun, 22 Feb 2009 04:48:10 GMT

I, along with a whole heck of a lot of other people will be attending PyCon in march. You should know about this by now, unless you’re living under a rock, or in a shoebox (I like shoeboxes).

PyCon 09 is turning out to be one of the ones I am most excited about in some time - barring the fact they let me actually stand up and speak about something, there’s a ton of other excellent and exciting things going on.

I will be doing two talks - “Introduction to Multiprocessing in Python” on Friday, at 3:20 PM, and “Concurrency and Distributed Computing with Python Today” at 3:20 PM Saturday.

The former talk is easy, given it will be focused on introducing the multiprocessing module to the masses, and taking questions about it (be gentle, I just work here). The latter is a much bigger beast. I’ve been blogging about my research in my pycon 2009 category, and I still have a pile of things to keep adding to that. The talk will attempt to differentiate concurrency from distributed systems, and show the various toolkits/frameworks/etc in the ecosystem today, to help you build both types of systems.

Given both talks are 45 minutes in length, I will be publishing my talk notes (my slides are not going to be heavy weight) and other information here.

In addition to me speaking, which may or may not be exciting, there’s one helluva ton of other talks which simply look awesome.

My schedule looks like this:

Thursday: Python Language Summit, where I will endeavor to be smart.
Friday: How to give a python talk
Friday: Using Windmill
Friday: Introduction to Python Profiling or How Python is Developed, to harass Brett.
Friday: Panel - Python VMs
Friday: Building an Automated QA infrastructure using Open Source Tools
Friday: Twisted, AMQP and Thrift: Bridging messaging and RPC for building scalable distributed applications
Friday: My Talk (I figure I should go to it)
Friday: A Whirlwind Excursion through Writing a C Extension or Challenges and Opportunities for Python
Thursday: Plugins and monkeypatching: increasing flexibility, dealing with inflexibility
Thursday: The (lack of) design patterns in Python or the Pinax talk
Thursday: Class Decorators: Radically Simple
Thursday: Panel: Object Relational Mappers: Philosophies and Design Decisions.
Thursday: Drop ACID and think about data (I wonder if we can take this literally, if he has scary slides though, we all might trip balls)
Thursday: My Talk, which is up against Bruce Eckel and Raymond H - I expect no one to show up.
Sunday: Panel: Functional Testing Tools in Python
Sunday: Designing a web framework: Django’s design decisions

This doesn’t even cover the open space discussions which I might attend - including the “Writing About Python” one Doug Hellmann is putting together as well as the “Teach Me Web Testing” one by Steve Holden.

After the main conference, I’ll be sticking around until Thursday morning for the sprints, at which point I should be sufficiently burned out on python stuff, I will fully convert over to being a full time burger flipper.

Right now there are 620 registered people who made their attendance public I don’t know how many there are in total.

Hope to see you there!

"Writing About Python" at PyCon

Fri, 20 Feb 2009 20:49:48 GMT

The Python writing community has never been stronger. From blogs to magazines to a slew of recent releases from O'Reilly, Packt, Manning, Apress, and other publishers there a more opportunities than ever before for anyone wanting to write about Python in some form.

Of course, authoring is only one aspect of producing high quality professional writing. As an editor, it's no surprise that I think all authors can benefit from having someone else read their work. Even an informal review can help identify gaps in an explanation or problems with flow, not to mention typos.

Writers, reviewers, and editors can all benefit from sharing tips, offering advice on tricky problem passages, or just chatting about ongoing projects. That list sounds just like the sorts of things we talk about when discussing programming, doesn't it?

I'd like to bring people together in person for a group discussion and a chance to network. To kick things off, I'm working on organizing a "meetup" as an open space session at PyCon in March. If you are a blogger, published writer, reviewer, or editor -- or aspire to be -- I want you to come and participate. No prior experience is necessary, everyone is welcome.

Let's connect people interested in doing technical reviews with authors and editors. Let's introduce potential writers to the editors and publishers. Let's learn from one another and improve our prose, the same way we improve our code.

We'll have to wait to schedule a precise time until the conference, but right now I'm thinking about late Saturday morning or early afternoon. If you are interested in attending, please post a comment so I can gauge the size of the room we'll need. If you have a preference for the time, make sure to include that information. And if you have ideas for things we should talk about, let's hear them.

Update: Although I was thinking of "print" writing, other forms of publishing like podcasting are welcome, too. We need more good Python podcasts!

Good code coverage: Necessity vs Sufficiency

Thu, 19 Feb 2009 14:56:02 GMT

I get really frustrated with posts that claim your unit tests lie to you or 100% code coverage is fallacious or there are flaws in coverage measurement. These are sensationalist headlines that encourage bad behavior, by confusing new or inexperienced or argumentative or lazy developers: "well, we all know test coverage isn't that important..."

There is a fundamental confusion here between necessary and sufficient. In plain English, things that are necessary are, well, necessary for a process; things that are sufficient are, collectively, sufficient.

One does not imply the other. Tires are necessary but not sufficient for a car.

Why do I bring this up? Because near-100% code coverage is basically necessary for a complete automated test suite, but not sufficient.

If you do not have near-100% code coverage, then some portion of your code is not being executed by your tests. That means that code is untested. It can't possibly be tested if you aren't executing it!

(As a corollary, no test that adds to code coverage is useless. It may not be as useful as it could be, but it is still running code that is otherwise untested. Room for improvement does not equal uselessness!)

OTOH, there are many reasons that 100% code coverage isn't sufficient to make your test suite good. For one, code coverage isn't branch coverage. For another, code coverage can't tell you if your feature set meets your client's needs or is in fact functional -- all it can do is tell you that your feature set meets your expectations at the moment.

So, let me suggest this simple pair of statements:

High (statement) code coverage in your tests is important.
High code coverage doesn't guarantee that your tests are complete.

I feel like high code coverage is actually a pretty low bar for a project. Whether or not you're using TDD, if you don't have high code coverage numbers, you're not really testing very thoroughly; it's generally pretty easy to amp up your code coverage numbers quickly, although writing the test fixtures can be a problem. I'll talk a bit about this in my PyCon talk.

Regardless of your tests suite, your project will probably fail in a boring and project-specific way.

--titus

P.S. Many people do get it, which is nice.

simplejson 2.0.9

Thu, 19 Feb 2009 09:00:57 GMT

simplejson (documentation) is a simple, fast, complete, correct and extensible JSON (RFC 4627) encoder/decoder for Python 2.4+. It is pure Python code with no dependencies, but features an optional C extension for speed-ups.

simplejson 2.0.9 is a major bug-fix update:

Adds cyclic GC to the Encoder and Scanner speedups, which could've caused uncollectible cycles in some cases when using custom parser or encoder functions

The History of Python: Adding Support for User-defined Classes

Thu, 19 Feb 2009 08:00:16 GMT

The History of Python: Adding Support for User-defined Classes. Guido designed the run-time representation first, and tried to design the syntax to include as few new parsing concepts as possible. The origins of explicit self are also explained.

DB2 support for Django is coming

Thu, 19 Feb 2009 07:58:50 GMT

DB2 support for Django is coming. From IBM, under the Apache 2.0 License. I’m not sure if this makes it hard to bundle it with the rest of Django, which uses the BSD license.

Creating a web framework with WSGI video

Wed, 18 Feb 2009 11:54:14 GMT

Creating a web framework with WSGI on Vimeo

The Michigan Python Users Group (MichiPUG) meeting topic from February 2009, presented by Kevin Dangoor. This screencast video shows us using WSGI components to build up a web framework piece-by-piece.

By the way, for people who are interested in working on the Bespin server, this is the kind of “web framework” that the server is built upon.

Announcing django-viewtools

Wed, 18 Feb 2009 06:35:31 GMT

Announcing django-viewtools. A really excellent idea—run ./manage.py viewtools --pdb /path/on/site/ to debug a view in your Django project that is raising an error using the Python debugger, or use --profile to run the full request cycle for that URL through the profiler.

CloudMade: A Summary of the Future of Mapping

Tue, 17 Feb 2009 20:25:53 GMT

CloudMade: A Summary of the Future of Mapping. CloudMade are now offering commercially supported APIs on top of OpenStreetMap, including geocoding, routing and tile access libraries in Python/Ruby/Java and a very neat theming tool that lets you design your own map styles. This is really going to kick innovation around OpenStreetMap up a notch.

Dulwich

Tue, 17 Feb 2009 07:27:56 GMT

Dulwich. A pure Python implementation of the Git file format and protocols. Reinforces my impression that a key to Git’s success is stable, well designed and documented on-disk formats.

Write to a Google Spreadsheet from a Python script

Tue, 17 Feb 2009 06:02:06 GMT

Write to a Google Spreadsheet from a Python script. I didn’t know Google Spreadsheets could directly serve dynamic images that automatically update when the underlying data changes.

Django Tech Blog

Tue, 17 Feb 2009 05:32:37 GMT

The new Django blog system I have been working on is virtually finished, I’m just testing it at the moment. Now I know that by writing my own blog system I will miss out on all the capabilities of a full featured and mature system, such as Wordpress; but my new system — which I call Django Techblog — has a few features that Wordpress doesn’t, but more importantly, it works just the way I want it to.

One of those new features is the ability to insert syntax highlighted code in to comments, which is very useful for a techy blog! I’ve created a post on the beta site, and I’d like to invite you to pop over and test the commenting system. I’ll delete the comments when I’m satisfied it is working nicely and I can replace www.willmcgugan.com. So feel free to play about with it…

http://beta.willmcgugan.com/2009/2/16/django-techblog-comment-test/

Any comments about the design or general functionality of the blog would also be welcome!

Explaining Why Interfaces Are Great

Mon, 16 Feb 2009 22:40:20 GMT

Why?

Why use interfaces? Especially with Python's new ABCs, is there really a use for them?

Some of us Zope Interface fans — names withheld to protect the guilty, although you may feel free to unmask yourselves in the comments if you like — have expressed frustration that ABCs beat out interfaces for inclusion of the standard library. However, I recently explored various mailing lists, Interfaces literature, and blogs, and haven't found a coherent description of why one would prefer interfaces over ABCs. It's no surprise that Zope's interface package is poorly understood, given that nobody has even explained it! In fact, PEP 3119 specifically says:

For now, I'll leave it to proponents of Interfaces to explain why Interfaces are better.

It seems that nobody has taken up the challenge.

I remember Jim Fulton trying to explain this to me many years ago, at a PyCon in Washington DC. I definitely didn't understand it then. I was reluctant to replace the crappy little interfaces system in Twisted at the time with something big and complicated-looking. Luckily the other Twisted committers prevailed upon me, and Zope Interface has saved us from maintaining that misdesigned and semi-functional mess.

During that explanation, I remember that Jim kept saying that interfaces provided a model to "reason about intent". At the time I didn't understand why you'd want to reason about intent in code. Wouldn't the docstrings and the implementation specify the intent clearly enough? Now, I can see exactly what he's talking about and I use the features he was referring to all the time. I don't know how I'd write large Python programs without them.

Caveat

This isn't a rant against ABCs. I think ABCs are mostly pretty good, certainly an improvement over what was (or rather, wasn't) there before. ABCs provide things that Interfaces don't, like the new @abstractmethod and @abstractproperty decorators. Plus, one of the irritating things about using zope.interface is that the metadata about standard objects in zope.interface.common is not hooked up to anything: IMapping.providedBy({}) returns False. ABCs will provide that metadata in the standard library, making zope.interface that much more useful once it has been upgraded to understand the declarations that the collections and numbers modules provide.

So, on to the main event: what do Zope Interfaces provide which makes them so great?

Clarity

Let's say we have an idea of something called a "vehicle". We can represent it as one of two things: a real base class (Vehicle), an ABC (AVehicle) or an Interface (IVehicle).

There are a set of operations that interfaces and base-classes share. We can ask, "is this thing I have a vehicle"? In the base-class case we spell that 'if isinstance(something, Vehicle)'. In the interfaces case, we say 'if IVehicle.providedBy(something)'. We can ask, "will instances of this type be a vehicle?". For an interface, we say 'if IVehicle.implementedBy(Something)', and for a base class we say 'issubclass(Something, Vehicle)'. With the new hooks provided by the ABCs in 2.6 and 3.0, these are almost equivalent. With zope.interface, you can subclass InterfaceClass and write your own providedBy method. With the ABC system, you subclass type and implement __instancecheck__.

However, there are some questions you can't quite cleanly ask of the ABC system. For one thing, what does it really mean to be a Vehicle? If you are looking at AVehicle, you can't tell the difference between implementation details and the specification of the interface. You can use dir() and ignore a few of the usual suspects — __doc__, __module__, __name__, _abc_negative_cache_version — but what about the quirkier bits? Metaclasses, inherited attributes, and so on? There's probably some way to do it, but I certainly can't figure it out quickly enough to include in this article. In other words, types have two jobs: they might be ABCs, or they might be types, or they might be both, and it's impossible to separate those responsibilities.

With an Interface, this question is a lot easier to ask. For a quick look, list(IVehicle) will give a complete list of all the attributes expected of a vehicle, as strings. If you want more detail, IVehicle.namesAndDescriptions() and Method.getSignatureInfo() will oblige.

Since the interface encapsulates only what an object is supposed to be, and no functionality of its own, it's possible for frameworks to inspect them and provide much nicer error messages when objects don't match their expectations. zope.interface.verifyClass and zope.interface.verifyObject can tell you, both for error-reporting and unit-testing purposes, whether an object looks like a proper vehicle or not, without actually trying to drive it around.

Flexibility

At the most basic level, interfaces are more flexible because they are objects. ABCs aren't objects, at least in the message-passing smalltalk sense; they are a collection of top-level functions and some rules about how those functions apply to types. If you want to change the answer to isinstance(), you need to register a type by using ABCMeta.register or overriding __instancecheck__ on a real subclass of type. If you want to change the answer to providedBy, for example for a unit test, all you need is an object with a providedBy method.

Of course, you can do it "for real" with an InterfaceClass, but you don't need to. In other words, its semantics are those of a normal method call.

Interfaces aren't completely self-contained, of course: there are top-level functions that operate on interfaces, like verifyObject. However, there's an interface to describe what is expected:

>>> from zope.interface.interfaces import Interface, IInterface
>>> IInterface.providedBy(Interface)
True

There's also the issue of who implements what. For example, you might have a plug-in system which requires modules to implement some functionality. Generally speaking, modules are instances of ModuleType, so specifying that all modules implement some type with an ABC is somewhat awkward. With an interface, however, there is a specific facility for this: you put a moduleProvides(IVehicle) declaration at the top of your module.

In zope.interface, there is a very clear conceptual break between implements and provides. A module may provide an interface — i.e. be an object which satisfies that interface at runtime — without there being any object that implements that interface — i.e. is a type whose instances automatically provide it. This distinction comes in handy when avoiding certain things. This distinction exists with ABCs; either you "are a subclass of" a type or you "are an instance of" a type, but the language around it is more awkward and vague, especially since you can be a "virtual instance" or "virtual subclass" now as well.

There's also the issue of dynamic proxies. If you have a wrapper which provides security around another object, or transparent remote access to another object, or records method calls (and so on) the wrapper really wants to say that it provides the interfaces provided by the object it is wrapping, but the wrapper type does not implement those interfaces. In other words, different instances of your wrapper class will actually provide different interfaces. With zope.interface you can declare this via the directlyProvides declaration. With ABCs, this is not generally possible because ABCMeta.register will only work on a type.

Adaptation

Let's say I have an object that provides IVehicle. I want to display it somehow — and in today's web-centric world, that probably means "I want to generate some HTML". How do I get from here to there? ABCs don't provide an answer to that question. Interfaces don't do that directly either, but they do provide a mechanism which allows you to provide an answer: you can adapt from one interface to another.

I'm not going to get into the intricacies of exactly how adaptation works in zope.interface, since it isn't important to understand most of the time. Suffice it to say you can adapt based on specific hooks that are registered, based on the type an object is, or based on what interfaces it provides.

The gist of it is that you have some thing that you don't know what it is, and you want an object that provides IHTMLRenderer. The way you express that intent is:

renderer = IHTMLRenderer(someObject)

If there are no rules for adapting an object like the one you have passed to an IHTMLRenderer, then you will get an exception - which is all that will happen, normally. However, this point of separation between the contract that your code expects and the concrete type that your code ends up actually talking to can be very useful.

The larger Zope application server has a rich and complex set of tools for defining which adapter is appropriate in which context, but Twisted has a very simple interface to adaptation. You simply register an adapter, which is a 1-argument callable that takes an object that conforms to some interface or is an instance of some class, and returns an object that provides another interface. Here's how you do it:

    from twisted.python.components import registerAdapter
    class VehicleRenderer(object):
        "Render a vehicle as HTML"
        implements(IHTMLRenderer)
        def __init__(self, vehicle):
            self.vehicle = vehicle
        def renderHTML(self):
            return "<h1>A Great Vehicle %s (%s)</h1>" % (
                       self.vehicle.make.name,
                       self.vehicle.model.name)
    registerAdapter(VehicleRenderer, IVehicle, IHTMLRenderer)

Now, whenever you do IHTMLRenderer(someVehicle), you'll get a VehicleRenderer(someVehicle).

Your code for rendering now doesn't need any special-case knowledge about particular types. It is written to an interface, and it's very easy to figure out which one; it says "IHTMLRenderer" right there. It's also easy to find implementors of that interface; just grep for "implements.*IHTMLRenderer" or similar. Or, use pydoctor and look at the "known implementations" section for the interface in question.

Conclusion

In a super-dynamic language like Python, you don't need a system for explicit abstract interfaces. No compiler is going to shoot you for calling a 'foo' method. But, formal interface definitions serve many purposes. They can function as documentation, as a touchstone for code which wants to clearly report programming errors ("warning: MyWidget claims to implement IWidget, but doesn't implement a 'doWidgetStuff' method"), and a mechanism for indirection when you know what contract your code wants but you don't know what implementation will necessarily satisfy it (adaptation).

Even with a standard library mechanism for doing some of these things, Zope Interface remains a highly useful library, and if you are working on a large Python system you should consider augmenting its organization and documentation with this excellent tool.

simplejson 2.0.8

Mon, 16 Feb 2009 09:56:05 GMT

simplejson 2.0.8 is a minor bug-fix update:

Documentation fixes
Fixes encoding True and False as keys
Fixes checking for True and False by identity for several parameters

PyMOTW: grp

Mon, 16 Feb 2009 00:19:24 GMT

grp – Unix Group Database

Purpose:	Read group data from Unix group database.
Python Version:	1.4 and later

The grp module can be used to read information about Unix groups from the group database (usually /etc/group). The read-only interface returns tuple-like objects with named attributes for the standard fields of a group record.

Index	Attribute	Meaning
0	gr_name	Name
1	gr_passwd	Password, if any (encrypted)
2	gr_gid	Numerical id (integer)
3	gr_mem	Names of group members

The name and password values are both strings, the GID is an integer, and the members are reported as a list of strings.

Querying All Groups

Suppose you need to print a report of all of the “real” groups on a system, including their members (for our purposes, “real” is defined as having a name not starting with “_”). To load the entire password database, you would use getgrall(). The return value is a list with an undefined order, so you probably want to sort it before printing the report.

import grp
import operator

# Load all of the user data, sorted by username
all_groups = grp.getgrall()
interesting_groups = sorted((g 
                            for g in all_groups 
                            if not g.gr_name.startswith('_')),
                            key=operator.attrgetter('gr_name'))

# Find the longest length for the name
name_length = max(len(g.gr_name) for g in interesting_groups) + 1

# Print report headers
fmt = '%-*s %4s %10s %s'
print fmt % (name_length, 'Name', 
             'GID', 
             'Password',
             'Members')
print '-' * name_length, '----', '-' * 10, '-' * 30

# Print the data
for g in interesting_groups:
    print fmt % (name_length, g.gr_name, 
                 g.gr_gid, 
                 g.gr_passwd,
                 ', '.join(g.gr_mem))

$ python grp_getgrall.py
Name                                      GID   Password Members
---------------------------------------- ---- ---------- ------------------------------
accessibility                              90          *
admin                                      80          * root, dhellmann
authedusers                                50
bin                                         7          *
certusers                                  29          * root, _jabber, _postfix, _cyrus, _calendar
com.apple.access_screensharing-disabled   101            dhellmann
consoleusers                               53
daemon                                      1          * root
dhellmann                                 501
dialer                                     68          *
everyone                                   12
group                                      16
interactusers                              51
kmem                                        2          * root
localaccounts                              61
mail                                        6          *
netaccounts                                62
netusers                                   52
network                                    69          *
nobody                                     -2          *
nogroup                                    -1          *
operator                                    5          * root
owner                                      10
postgres                                  401
procmod                                     9          * root
procview                                    8          * root
racemi                                    500            dhellmann
smmsp                                      25
staff                                      20          * root, test
sys                                         3          * root
tty                                         4          * root
utmp                                       45          *
wheel                                       0          * root

Group Memberships for a User

Another common task might be to print a list of all the groups for a given user:

import grp

username = 'dhellmann'
groups = [g.gr_name for g in grp.getgrall() if username in g.gr_mem]
print username, 'belongs to:', ', '.join(groups)

$ python grp_groups_for_user.py
dhellmann belongs to: _lpadmin, admin, com.apple.access_screensharing-disabled, racemi

Finding a Group By Name

As with pwd, it is also possible to query for information about a specific group, either by name or numeric id.

import grp

name = 'admin'
info = grp.getgrnam(name)
print 'Name    :', info.gr_name
print 'GID     :', info.gr_gid
print 'Password:', info.gr_passwd
print 'Members :', ', '.join(info.gr_mem)

$ python grp_getgrnam.py
Name    : admin
GID     : 80
Password: *
Members : root, dhellmann

Finding a Group by ID

To identify the group running the current process, combine getgrgid() with os.getgid().

import grp
import os

gid = os.getgid()
group_info = grp.getgrgid(gid)
print 'Currently running with GID=%s name=%s' % (gid, group_info.gr_name)

$ python grp_getgrgid_process.py
Currently running with GID=501 name=dhellmann

And to get the group name based on the permissions on a file, look up the group returned by os.stat().

import grp
import os
import sys

filename = 'grp_getgrgid_fileowner.py'
stat_info = os.stat(filename)
owner = grp.getgrgid(stat_info.st_gid).gr_name

print '%s is owned by %s (%s)' % (filename, owner, stat_info.st_gid)

$ python grp_getgrgid_fileowner.py
grp_getgrgid_fileowner.py is owned by dhellmann (501)

Embedding IronPython 2 and the Dynamic Language Runtime (turtles and articles)

Mon, 16 Feb 2009 00:11:40 GMT

One of the most impressive things about DLR based languages like IronPython and IronRuby is how easy they are to host in applications to add dynamic features. You can even host IronPython in IronPython, something that Resolver One makes good use of to provide a separate execution context for each spreadsheet. ... [355 words]

A better Caching system for Django

Sun, 15 Feb 2009 21:55:14 GMT

Django has pretty good support for caching, which is one of the easiest ways of speeding up a web application. The default is to cache for ten minutes, which means that if you get multiple requests for a page within a ten minute window then Django can serve up a stored copy of the page without hitting the database or rendering the HTML. The caching period can be set per-page and fragments of pages can be cached rather than the whole, but the system rests on the fact that it doesn’t matter if the content doesn’t change for a period of time.

Trouble is, not many web applications work that way. Consider a humble blog post with a piece of content and a list of comments. Granted, the post content isn’t going to change very often and will probably never change again once the author has corrected all the typos, but a list of comments may change at any point through the life-time of the blog. If the page were cached, comments would not be visible on the page for a short period of time, which doesn’t give that instant gratification that web users expect these days.

It would be nice if a web app could benefit from caching and still serve a fresh page when content changes. Alas, Django’s default time-based caching is never going to achieve this. What is needed is a way of invalidating an item in the cache when there are changes that alter the content; using the blog example, this would be when the post content changes or a new comment is submitted. Django does give you the tools to do this — using the low level cache API, you can construct a cache key (a simple string) that changes according to the page content, then store and retrieve your content manually. For example you might create a cache key with the modification date of the post, and the ID of the most recent comment. Then the page need only be rendered when the cache key isn’t found in the cache. The downside of this approach is that it can take a little work to calculate the key (you may still have to hit the database to generate the cache key). This is not quite as nice as time-based caching, which will do negligible work for cached pages.

A better approach, which has the benefit of always serving a fresh page, and still requires negligible work for cached pages is to construct the cache key based on the request, then invalidate (i.e. delete) the cache key when the content changes. Again, this can be done with Django, although it has no explicit support for it. Such a system requires more work because the web app must track any event that could potentially require a fresh page to be generated, but has the major advantage that up-to-date pages are virtually always going to be available in the cache.

I’ve written ad-hoc page-event cache systems a couple of times and they are a major headache to maintain. Even on simple pages there can be multiple events that may invalidate a cached page. What would be nice, is some formal way of associating a model instance with a the URL(s) that depend on it, so that when the instance is saved, the associated pages will be invalidated and re-generated at the next request.

I’ve not yet settled on the best way of implementing this, but it will probably be a decorator that builds up a mapping of object type plus primary key to its dependant URLs, and a mixin for models that hooks in to the save method.

I’m open to suggestions regarding the most elegant implementation, and any other potential solutions!

Bespin: code in the cloud

Sun, 15 Feb 2009 10:30:19 GMT

Despite working for an open source company, I have been pretty quiet here about what I’ve been doing in the Mozilla Labs web developer tools group. No more. We’ve gone public!

Mozilla Labs » Blog Archive » Introducing Bespin

Bespin proposes an open extensible web-based framework for code editing that aims to increase developer productivity, enable compelling user experiences, and promote the use of open standards.

I started working on Bespin as soon as I joined Mozilla, hitting the ground running with a new Python server. Ben and Dion had already done a lot of work and experimentation on Bespin prior to joining Mozilla in December, so I must confess that I am still fairly ignorant about the Canvas-based magic that they’re doing in the UI. But, Bespin has an architecture that lends itself well to selective ignorance: the server provides a RESTful API, and the client is responsible for all of the presentation. For their part, Ben and Dion have been able to be blissfully ignorant about the inner workings of the Python server.

Of course, I’m not a JavaScript noob and have done some work in the client, but my focus has been the server. Now that we’re out in the open, you can definitely expect that we’ll be talking more about how things work and how you can bend Bespin to your will. Bespin is honest to goodness open source (MPL-licensed), so it becomes an open and collaborative effort starting right away.

The initial reaction has been fantastic. There are tons of people hanging out in #bespin on irc.mozilla.org, and the mailing list has grown to a couple hundred members already. Thanks to everyone for jumping in with your thoughts and patches!

Here’s some of the coverage:

Dion’s post at Ajaxian:

From Dion’s personal blog:

Foolish chaps and companies have come to me in the past thinking that open source will be a silver bullet for “getting other people to do our work.” Those that have been involved in open source know that it isn’t the case. It is often more work. But, it is worth it.

From Ars Technica:

The project is still at an early stage of development and there is clearly a lot of work to be done before it will be able to deliver the same practical value as existing desktop editors. Despite the limitations, it shows an enormous amount of promise and has the potential to eventually deliver a user experience that rivals even the best text editors.

From Five Questions with Dion Almaer:

Now the browsers are moving fast again and building a first class platform for us to develop, the Open Web Platform. Instead of getting bogged down thinking about what IE 6 gives you, take some time to think about what you could build with the latest technology. I realise that you have to be pragmatic and get things working with your audience, but browsers are changing, and so are expectations.

From What Mozilla’s Bespin Bespeaks (ComputerworldUK):

You can see that Bespin is ticking all the Mozilla boxes, but what’s also striking is that this is a Web-based project: Mozilla is entering the cloud. It’s a further shift to viewing the Web as a platform for doing, well, just about anything. Clearly, against that background, open standards are even more important. And not only for the code: another issue that Mozilla will need to start addressing publicly is that of open data. As more stuff moves into the cloud, it become imperative to establish minimum standards for access to the data that is held there. I look forward to hearing Mozilla’s views on the subject.

While I certainly don’t speak for Mozilla, I would be extremely surprised if there’s anyone at Mozilla that believes that users should have anything less than full access and ability to take their data with them. There can be technical issues involved in providing the data, but the data should be available in some reasonable form. Bespin, for its part, makes it easy to export a project in a tarball or zipfile.

I was surprised to see Bespin covered even on Lifehacker:

Primarily, Bespin is a text editor—the kind you’d use for editing code or managing text-based todos. Using Bespin, developers could collaborate on projects through a unified interface (that still supports plugins!) no matter where they are—so long as they’ve got a browser.

cnet has the story, too:

For example, what about integration with open-source software repositories? If it’s flexible enough, Bespin could essentially act as a source code viewer that repositories such as SourceForge or Google Code could employ.

A nice writeup on the ReadWriteWeb as well:

It’s clear that a great deal of thought and attention went into this early version - and it’s a safe bet that it will only get more impressive as time goes on.

RWW last month surprised me with their coverage of me joining Mozilla.

I’m having a great time at Mozilla so far, and it’s great to be out in the open working with so many people now on Bespin and ServerJS.