Twisted Matrix Laboratories

Fluidinfo sponsors Twisted!

2010-08-07T20:11:00.000-07:00

Fluidinfo is Twisted's latest sponsor! We asked if they could share some of the joys of sponsorship with us, and they had this to say about it:

At Fluidinfo, we're heavy users of Twisted. All of our infrastructure depends on it. FluidDB, our social database, is entirely built on Twisted, and we've released several core parts of it as open source: txAMQP, txRDQ and txThrift. Not only that, but we have contributed to Twisted both with code and (albeit small) personal donations.

It's not just that sponsoring was the fair the thing to do, it has also produced tremendous results in a framework crucial to our business. When you donate to the TSF it's simple math that more bugs get fixed, but also the quality of the entire Twisted project is enhanced. And when that happens, all of our products are enhanced automatically without us writing one line of code. So sponsoring is not just an act of generosity, it's an investment in Fluidinfo. Using and sponsoring Twisted has been an indispensable "force multiplier" for a growing start-up like us.

– Esteve Fernandez

CTO, Fluidinfo Inc.

Twisted 10.1.0 released

2010-07-04T08:42:00.000-07:00

Twisted 10.1.0 is finally out. It's a month late, but it's finally out there and free from the shackles of British rule.

Highlights include:

Deferreds now support cancellation
A new "endpoint" interface which can abstractly describe stream transport endpoints such as TCP and SSL
inotify support for Linux, which allows monitoring of file system events.
AMP supports transferring timestamps

Note also that this is the last release in which we will support Windows for Python 2.4.

For more information, see the NEWS file.

It's stable, backwards compatible, well tested and in every way an improvement. Download it the tarball, the Windows installer for Python 2.5 or the Windows installer for Python 2.6.

Update: We now have executable Windows installers for Python 2.5 and Python 2.6 to complement the MSIs.

Many thanks to Glyph Lefkowitz, who helped prepare the release, and the PyCon 2010 sprinters, who did so much of the work for this release.

Twisted 10.1.0pre1 released – please test

2010-06-19T05:47:00.000-07:00

From the dank and grey depths of a London summer, a thin, reedy voice wanders across the waves of the Internet, saying that it's jolly well pleased to announce the first 10.1.0 pre-release.

We would very much appreciate it if you could download the tarballs, test them on your systems, and file tickets as appropriate.

Highlights include:

Deferreds now support cancellation
A new "endpoint" interface which can abstractly describe stream transport endpoints such as TCP and SSL
inotify support for Linux, which allows monitoring of file system events.
AMP supports transferring timestamps

For more information, see the NEWS file.

If there is only positive feedback from the pre-release, we aim to do the final release some time later this month.

Powering the Silver Screen

2010-05-27T08:52:00.000-07:00

Think Twisted is only good for Internet-facing servers? Even then, it's not like Python itself that touches every part of your day in some way, right? Well my friend, you'd be delightfully wrong.

When you watch a movie crediting Lucasfilm for special effects, Twisted could be behind the scenes. Since 2004, Lucasfilm has used Twisted as part of its rendering operations to proxy database connectivity in the render farm (and much more), 24/7 x 365. That's right my friends, Twisted is not only the engine of your Internet...it might have a hand in your entertainment too.

To read more check out the great Success Story from Dave Peticolas from Lucasfilm: Twisted at Lucasfilm

Have your own success story using Twisted? Let us know! Just send an e-mail to success@twistedmatrix.com

Twisted Git (and Bzr) Mirror

2010-05-20T09:25:00.000-07:00

With the multitude of distributed version control systems now available, it's getting pretty hard for any open source project to get by on a centralized system. Distributed systems like bzr, mercurial, or git give equal tools to all contributors, not just a select few trusted to vette changes for inclusion in an official branch. Forcing new contributors to work with diff and patch is just plain mean!

So, while Twisted's source is still hosted in subversion, and likely will be for some time to come (due to various toolchain requirements), I'm happy to announce that there is now a git mirror to complement the existing bzr mirror.

With these mirrors available, developers who don't have commit access to the subversion server can still work with Twisted using modern revision control tools - branching, merging, history, and all the rest.

The workflow for merging changes based on the bzr or git mirror back into Twisted trunk isn't yet completely worked out. Likely it won't be until people start using these and trying to push changes back to us. So if you're a fan of one of these two systems, please, start using these mirrors and help us figure out the best way to integrate things back into subversion.

Twisted test suite code coverage

2010-04-25T09:07:00.000-07:00

As anyone who's contributed to Twisted knows, we currently have a very high standard for code changes and additions. For a long time, we've relied mainly on the standard library trace module. The trace module is quite serviceable, but recently some other options have come along which are much better. In particular, coverage.py seems to provide enough additional features that it's actually worth integrating into Twisted's development process. For example, it has some basic branch coverage features (trace will show you what lines were executed; coverage.py will show you this, plus tell you whether both possible outcomes of your "if" statement were taken), and can generate fairly nice HTML reports.

So, as a first step, I've set up a coverage.py builder on Twisted's BuildBot. This means that code coverage reports are part of Twisted's continuous integration system now: for every trunk revision, new coverage reports will be generated. At the moment, these reports end up jumbled up in the builds directory along with a lot of other stuff. For example, here's the latest report as of this post. Something I still want to do is clean up how these reports are organized so they're easier to find and compare.

As you can see from this report, Twisted has 86% line coverage and 67% branch coverage (which I had to compute myself from the totals at the bottom of the page - one of a few coverage.py nits that I hope will be fixed soon :). coverage.py decides this makes overall coverage 83%, though I'm not sure how exactly it arrives at that number.

That's pretty good, but there's plenty of room for improvement. I hope these reports will become an easy way for people to find areas of Twisted which need more coverage and motivate them to contribute new tests.

Oh, and since coverage.py only reports on Python coverage, I've also set up a gcov/lcov builder which uploads its coverage reports for execution of the C code in Twisted to a similar location.

New, New, New!

2010-03-30T20:12:00.000-07:00

New Success Story

We've added a great success story for PowerCard, a loyalty rewards program for local restaurants. One of the terrific things about the PowerCard story is the use of Twisted for both back-end and client-facing systems. It's a great reminder that Twisted is as useful solving problems on Windows platforms as it is on the Linux/UNIX-based systems many of us code for.

Read the full success story here.

New Library Listing

Looking to use Twisted with new technologies like RabbitMQ (txAMQP) and Redis (txRedis)? Browse the updated libraries listing. We bet you'll find what you're looking for.

New Twisted Ambassador

And why do we have these new and updated listings for 3rd-party code? It's because we have a new guy who has volunteered to maintain and update the listings!

Jason J. W. Williams is Twisted's new ambassador to 3rd-party projects of all kinds.Primarily this responsibility involves keeping the "Success Stories" and "Projects Using Twisted" pages up to date, and occasionally writing small updates (like this one) for the news feed.

Please contact success at twisted matrix dot com if you have a success story about how you used Twisted, a pointer to some hitherto unknown code that uses Twisted, or for more information. And watch this space for more updates!

Barcelona Twisted Sprint

2010-03-22T02:38:00.000-07:00

Yesterday's Twisted sprint at the Flumotion office was quite a success. There were six of us there, and many first time Twisted contributors.

Jan Urbański worked on fixing a couple IMAP4 client search bugs. He also spent some time using psycopg's asynchronous interface to create a very Twisted-friendly postgres API, with methods that return Deferreds, and all without using any threads.

Thomas Vander Stichele and Zaheer Merali worked on addressing ambiguities on the AMP API documentation and adding support for dates and decimals to AMP.

Later Zaheer switched to pairing with Sebastien Merle who was working on adding string streaming to AMP. Meanwhile Thomas worked with Andoni Morales (who had been pairing with Sebastien) on improving the idle behavior of Twisted's glib2 integration.

Esteve Fernandez worked on adding streaming upload support to Twisted Web's resource model.

Jessica McKellar helped out remotely with reviews of fixes for Windows child process issues, Conch example bugs, Windows+Python 2.7 compatibility, and release-related documentation errors.

Kevin Horn also worked remotely on some problems with the structure of our documentation.

And I walked around and answered questions about Twisted development and made suggestions about implementation approaches to try out. I also reviewed and merged the AMP documentation fix, and also worked on another IMAP4 ticket, improving test coverage for the client's SELECT support.

Twisted 10.0.0 released

2010-03-03T15:26:00.001-08:00

On behalf of Twisted Matrix Laboratories, I am honored to announce the release of Twisted 10.0.

Highlights include:

Improved documentation, including "Twisted Web in 60 seconds"
Faster Perspective Broker applications
A new Windows installer that ships without zope.interface
Twisted no longer supports Python 2.3
Over one hundred closed tickets

For more information, see the NEWS file.

It's stable, backwards compatible, well tested and in every way an improvement. Download the tarball, the Windows installer for Python 2.5 or the Windows installer for Python 2.6.

Many thanks to Jean-Paul Calderone and Chris Armstrong, whose work on release automation tools and answers to numerous questions made this possible. Thanks also to the supporters of the Twisted Software Foundation and to the many contributors for this release.

PyCon 2010 sprint report

2010-03-02T08:04:00.000-08:00

Residents of the once-great city of Atlanta are only now beginning to piece their lives back together after the event that Glyph Lefkowitz described as "the best Twisted sprint ever".

Quite apart from the unprecedented levels of wanton destruction, Twisted's PyCon 2010 sprint was also better attended than any previous sprints, which have laid waste to the outskirts of Boston, inner-city Dallas and regions of Australia.

At its peak, there were eighteen people seated around the two tables where the sprint took place. People even joined in remotely, working with the Atlanta team through #twisted on freenode.

Although buried deep within a secret bunker in Atlanta, the sprint still managed to attract a fair number of new contributors and Twisted users who wanted to help out.

Attendees who made themselves known to your correspondent include:

Jonathan Lange

Jean-Paul Calderone

David Reid

Thomas Hervé

Tim Couper

Pavel Pergamenshchik

Maciej Fijalkowski

Glyph Lefkowitz

Itamar Turner-Trauring

Terry Jones

Wilfredo Sánchez Vega

Kevin Horn

Jessica McKellar, Ralph Meijer and Andrew Bennetts also participated remotely.

Attendees all had immense fun and expressed a desire to have another sprint soon. One observed that it had been far too long since giant mecha robot warriors descended on Paris.

Twisted prevails,
jml

More simply, we had a great time at the sprint and got a ridiculous amount of stuff done.

We fixed many, many bugs, including all of the known regressions since the 9.0 release. Once we fixed the regressions, we started work on getting the first 10.0 pre-release out the door.

The pre-release probably would not have been possible without the sprint since it allowed the new release manager ready access to Jean-Paul, who has quite a lot of knowledge stuck in his head. One result of this is a new release process document, which is being developed alongside the actual 10.0 release.

We also reached an in-principle agreement to aim for a release every three months. This means that although hardly any of the good work from the sprint is in the 10.0 release, you won't have to wait long for it to appear in 10.1. Hopefully. I plan write up how this will work as soon as the 10.0 release is done.

The sprint proved to be a perfect opportunity to talk about some of the more gutly bits of Twisted, including Deferred cancellation (see #990) and endpoints (see #1442). The former is done, and the latter has made good progress. Stay tuned.

Thomas Hervé did a lot of work on upgrading our Trac instance, which was getting kind of old and currently crashes all of the time.

We talked a bit about how we could make it easier for people to contribute to Twisted without compromising our high standards of quality. We put as many of the ideas that we could remember on to http://twistedmatrix.com/trac/wiki/ContributionIdeas.

Maciej Fijalkowski was at the sprint working on getting PyPy benchmarks for Twisted and on making PyPy work faster with Twisted. Personally, it's really exciting to see PyPy live up to its promise of a faster Python, particularly with a big codebase like Twisted.

As I said, we fixed lots of bugs. Here's the complete list:

#4179 - HTTP proxy hanging in Twisted 9.0

#3056 - twisted.spread.pb.IUsernameMD5Password's docstring should say it accepts an MD5 digest of a password, not plaintext

#3289 - startLoggingWithObserver raises IndexError

#4178 - Function singleSearchStep was renamed to _singleSearchStep not everywhere

#4192 - Move the content from exarkun's Twisted Web in 60 Seconds series into the Twisted Web howto

#4284 - t.e.adbapi leaks threads

#4282 - _release.filePathDelta is broken

#3718 - twisted.internet.posixbase._Waker not defined on Jython

#4143 - twisted.web.wsgi incorrectly unquotes QUERY_STRING

#4255 - BaseLogFile should provide a method to reopen logs

#972 - Add linux inotify support to twisted core

#2367 - twisted.test.test_failure.FailureTestCase.testBrokenStr break #trial in debug mode

#1083 - gtk2reactor thread initialization confuses python-2.3

#3462 - FTP server: upload should support async close

#990 - Deferred cancellation

#733 - twisted's SIGCHLD handler breaks popen.

#2376 - release-twisted should generate guides with links to API docs

#4272 - flavors.py still talks about getObjectAt

#4172 - UDP crashes python.exe when using IOCPReactor

#3925 - test_addresses fails with Glib2 and Gtk2 reactors

#4255 - BaseLogFile should provide a method to reopen logs

#4293 - Update links in testing documentation

#4311 - XMPP Stanza Error 'remote-server-timeout' yield no type and code

#3811 - Allow aborting HTTP client connections in the _newclient API

We also worked on:

Getting rid of 3k warnings and generally doing 2to3
Event support for iocpreactor
Making Deferreds more iterative (see #411)
Twisted Web features and bug fixes
Fixing obscure Windows/gtk2 compatibility issues

Thank you very much to PyCon US, the Python Software Foundation and the many sponsors of the Twisted Software Foundation for their hard-work and support in making this all possible.

Thanks also to the attendees for making it such a blast.

If you want to encourage more of this sort of behaviour, then please donate to the Twisted Software Foundation.

jml

Twisted 9 is in the works, first pre-release is available

2009-11-20T19:03:00.001-08:00

Hi all! I've done an initial pre-release of Twisted 9. I've made available a set of tarballs (The one called Twisted-9.0.0pre1.tar.bz2 is the main release) and the release notes.

Please try out this prerelease and tell us about any regressions or other release-related problems you find. If all goes well, we want to hear about that too.

New OS X build slave

2009-11-06T19:20:00.000-08:00

As some of you may recall, Apple lent Twisted an Xserve G5 a few years back. This was invaluable in maintaining Twisted support for OS X. A few weeks ago we unfortunately lost the ability to host this machine. Fortunately, David Reid came through and very generously donated a

which is now set up and running the Twisted test suite continuously. Thanks, David!

Canonical Men of Twistery

2009-05-25T02:44:00.000-07:00

The title of this quick little post goes to the excellent name mwh gave the linked photo :-) Taken at a dinner during AllHands in a gorgeous area outside of Barcelona, this pic shows the collection of Twisted developers that Canonical has hired (so far!).

Top row, left to right: Andrew Bennetts (spiv), Thomas Herve (therve), Chris Armstrong (radix), Michael Hudson (mwh), Jamu Kakar (jkakar).

Bottom row, left to right: Nicola Larosa (tekNico), Duncan McGreggor (oubiwann), Jonathan Lange (jml).

Pretty freakin' cool :-)

Sponsorship In 2009

2009-05-23T01:59:00.000-07:00

It's that time of year again; the time when you all have the excellent opportunity to settle your karmic debt and sponsor the Twisted project.

Actually, that time is every day of the year. Right now is just when we have arbitrarily decided to make you aware of this opportunity, so that we're not annoying you all year-round. Sometime soon we'll be taking down the current list of sponsors for 2008 — hopefully, to make room for a new list of sponsors for 2009. But without your help, that list will be empty!

Thanks to our founding sponsors, Twisted has had a great year. If you've been reading this news feed, you know that we've been able to review, respond to, and close tons of tickets this year.

As Jean-Paul Calderone previously wrote, this is the first year we've had a year-over-year decrease in the number of open tickets; and that's despite an increase in the number of filed tickets. We believe that Twisted is setting a great example for how other open source projects can effectively use sponsorship to fund development.

Many open source projects are perpetually crunched for volunteer labor. Twisted is no different in this respect — we could always use more help from the community. However, we are lucky to have something that most projects don't: the ability to convert dollars directly into maintenance that benefits the project.

When you contribute money to the Twisted project, you are directly contributing to get bugs fixed, features implemented, and tickets reviewed. Last year we told the same story, but this year we have an excellent track record. I invite you to read the history of our sponsored development — just scroll down and have a look. I'll wait.

... Done?

If you prefer visual representations of our success instead, consider these graphs. Here's a graph of all tickets reviewed this year, by the handle of the person who reviewed them:

Here's the same breakdown of all tickets that were closed this year:

In case you don't already know: "exarkun" is the handle for Jean-Paul Calderone. By his own calculation, at least 75% of his work this year has been sponsored development.

This isn't to say that Twisted is being supported entirely by sponsorship: as you can see from the huge number of tickets that exarkun has reviewed, a substantial proportion of development has been coming from the community, but sponsorship has allowed us to be more responsive to the community and integrate contributions much faster. (Developers may not review their own tickets, so there's no overlap between the code JP has been writing and the tickets he's been reviewing.)

This year of sponsorship has been great for the project. But, we're coming to the end of our first year of funding, and we need your help to keep going.

Every little bit helps, from small, one-time individual donations to monthly subscriptions to corporate sponsorships. This year, we need even more help from individuals, as some of our corporate sponsors have been impacted by the current economic climate and may not be able to contribute at the same level this year.

To summarize, there are a few things you can do to help the Twisted project:

Contribute code. Any contribution is welcome, but the best would be to attack an existing ticket in the tracker, especially one that has stalled on its way through review.
Donate money directly. You'll find donation buttons on the front page of twistedmatrix.com.
Donate money from your organization. Doing so gets you a voice on our sponsors-only mailing list, and can help you to set priorities for the coming year of development. You also get your logo on twistedmatrix.com, our gratitude, and last but not least, more Twisted development in the coming year!
Help us in our discussions with larger organizations. If you're a software developer using Twisted in a big company, we need your help to convince your organization that they should be contributing. Even if not, the board of the Twisted project is small and strapped for time - we could use some help just finding the right person to talk to, and making sure that all of their questions are answered promptly. (If you'd like to help with this effort, please contact me directly.)

Thanks for your interest in Twisted.

February Sponsored Twisted Development

2009-03-02T07:55:00.000-08:00

Another month, another round of Twisted development. February saw progress on a miscellany of bug fixes, as well as further work on the new SIP implementation. List of tickets I worked on (bold tickets are now closed):

#2036 - trial runs tests from .pyc files even if there is no .py file
#2194 - small bug in SIP Via header generation
#3634 - restat in render in static.File raise Exception if file doesn't exists
#3639 - suppress deprecation warnings emitted by twisted.test.test_sip
#3643 - MemCacheProtocol does not return an error if it is used after disconnecting
#3648 - twisted.cred.credentials.UsernameHashedPassword doesn't hash password strings when checkPassword
#3654 - unlink of unix socket in connectionLost may cause twistd daemons to ignore SIGTERM
#3584 - SIP transport and transaction layer
#3660 - IRCClient reports ":None:None" to CTCP version request

Work on #3654 unearthed a more general problem where twistd fails to exit when a service encounters an exception during shutdown. So hopefully there'll be less need to use kill -9 during development :)

RFC 3261 is the longest RFC for a network protocol the IETF has ever produced. The SIP transport/transaction branch only provides the bottom two layers of a full SIP stack, but introduces 1700 new lines of code. Special thanks to Chris Armstrong and Jean-Paul Calderone for their amazingly detailed review. I managed to get halfway through the issues they raised, and expect to finish it up next time.

Once again, thanks to the Software Freedom Conservancy, all of the Twisted Sponsors, and all the other developers who contribute to Twisted.

January Sprint Report

2009-02-01T00:08:00.001-08:00

Today, we had a last-minute unofficial Twisted sprint for January, at the Divmod office in Porter Square, Cambridge, Massachusetts.

Thanks to everyone who contributed.

Glyph Lefkowitz (that's me) reviewed 4 tickets, shortening the review queue by half.
Jean-Paul Calderone addressed review feedback for some of our old favorite tickets, compatibility with os.popen and lore's handling of whitespace.
Christopher Armstrong created a tool to generate the official API documentation, as well as reviewing Jean-Paul's lore work.
Itamar Shtull-Trauring helped with various things, including —
David Sturgis worked to fix twisted.words's IRC compatibility with Pidgin, among other IRC clients.

Thanks also to other contributors today not at the sprint: 'atsuoi' who contributed a patch for an SMTP issue, and Paul Swartz, who (it looks like) spent some time working on testing user authentication in Conch. (Commit those changes, Paul!)

We are tentatively planning to hold our next sprint on Sunday, February 22. If you're in the Cambridge, Massachusetts area, please stop by!

More January Sponsored Twisted Development

2009-01-26T15:18:00.000-08:00

First, a list of tickets I worked on:

#2036 - trial runs tests from .pyc files even if there is no .py file
#3575 - Common implementation of RFC 2617 digest authentication (closed)
#3582 - Improve SIP URI parsing/formatting
#3584 - SIP transport and transaction layer

It is the last of these on which I spent the bulk of the time, implementing the lower layers of SIP message handling as described in RFC 3261. As far as I know, this is the first implementation of this protocol in Python that has thorough unit test coverage! While reviewing the code from Sine, Divmod's SIP implementation, and comparing to the spec I discovered some significant misunderstandings that I had the chance to fix here. This code lays the foundation for both user agent code (like VoIP clients or SIMPLE instant messaging) and proxies.

Thanks to the SFC and all of the sponsors who made this possible, as well as to all the other Twisted developers who helped out by writing or reviewing code.

January Sponsored Twisted Development

2009-01-19T09:13:00.000-08:00

On Friday I finished the first two weeks of sponsored Twisted development for 2009. This week, the number of open tickets in the Twisted issue tracker hit 1040 - going down. We hit 1040 on the upside in November of 2007. Thanks in very large part of the sponsors who made so much of my time available for Twisted development in 2008, we closed more tickets than we opened last year for the first time (Drew P quipped that Twisted will be bug-free and feature-complete in 2636 - better than never ;). On average, we closed one ticket every 14 hours last year - all year. This trend is clearly visible in the graph included in the weekly summary of issue tracker activity. The green line shows total tickets filed, the red line shows total open tickets.

During this round of development, there was the usual smattering of web-related work:

#446 - Twisted Web accepts any HTTP method as if it was GET
#886 - Add an HTTP 1.1 client protocol implementation to twisted.web
#1176 - A t.web glossary is needed (closed)
#1522 - Remove woven from next release of twisted.web (closed)
#2446 - Python 2.4 has poor round-trip behaviour for xmlrpclib.DateTime; Twisted should work around
#2030 - Deprecate twisted.web.trp
#3035 - deprecate Resource subclasses in twisted.web.error, add equivalents in twisted.web.resource
#3416 - rename `twisted.web2.iweb.IOldRequest` to `twisted.web.iweb.IRequest` (closed)
#3457 - Session expiry check frequency should be based on sessionTimeout
#3600 - test twisted.web.domhelpers against xml.dom.minidom (closed)
#3601 - crash in twisted.web.http triggered by a blackberry web browser (closed)
#3605 - Factor "identity" transfer encoding support out of HTTPChannel.rawDataReceived into an object like the chunked transfer encoding object (closed)
#3610 - Give twisted.web.http_headers.Headers a nice repr (closed)

Of course #886 shows up in the list yet again. At this point the protocol implementation is essentially done. The code has been reviewed and most of the feedback dealt with. The only reason it isn't closed yet is that we want to provide more than just an HTTP/1.1 client protocol implementation. We want to provide an easy-to-use, high-level web client. Before I call #886 finished, I want to implement a few of those high-level APIs. So #886 is going to remain open for a little while longer, but I expect it to be closed very soon now.

Another highlight on this list is #1522 - the removal of Woven from Twisted. That's right, it's gone, guys. I really hope that after almost three years of being deprecated and almost six years of there being better alternatives around no one still has code using this. If so, you're out of luck for the next Twisted release. Perhaps you should take a look at Nevow. :)

Aside from web tickets, there were some more FTP improvements:

#3593 - passivePortRange in ftp.py not working (closed)
#3596 - Active mode makes ftp.py crash (closed)

Some improvements which make writing unit tests with Twisted and for Twisted-based code easier:

#3598 - TestCase.flushWarnings shouldn't use source files to implement its "offendingFunctions" feature (closed)
#3609 - Add IConsumer and IProducer support to StringTransport to facilitate testing producers and consumers (closed)

And all kinds of other assorted fixes and enhancements:

#733 - twisted's SIGCHLD handler breaks popen
#1785 - reactor.run should raise an error when called while the reactor is running (closed)
#3560 - remove usage of microdom from twisted.lore
#3576 - add high-level, cross-platform close-on-exit togglers to twisted.internet.fdesc (closed)
#3591 - ThreadedResolver--and twisted.internet.thread--assumes that the reactor is the installed reactor
#3595 - the interaction of the fireOnOneErrback and consumeErrors parameters to DeferredList is not well documented
#3602 - Improve factoring of various reactor implementations of "internal" FD tracking
#3606 - incorrect link in application howto (closed)

Changes to the internal factoring of reactors (#3602) and additions to fdesc (#3576) were both made in support of #733 - a very old ticket for a behavior which often trips people up. Glyph did the initial work on these improvements and Itamar and I came back and polished them a bit last week. With one of these closed and the other almost closed, Twisted is getting pretty close to not getting in the way of popen (and more generally, not causing syscalls to start to fail with EINTR). This is a generally nice thing, but it's taken a long time because it requires access to POSIX APIs Python doesn't expose.

Further work on #3560 has gotten Lore almost to the point where it's using minidom instead of microdom. Glyph raised a point in his review of the code which threw one further wrench into the workings, though. Currently, Lore input documents are implicitly XHTML. This means a Lore input document may use named character entities without declaring one of the XHTML DTDs. However, in order to continue to support this with minidom, it's necessary to reach way into expat and use a couple very confusing features. Itamar and I finally worked out how to do this, but if one is using xml.sax (which we are) then it involves using private attributes. Not nice. Tune in next month to learn how this turns out. ;)

Of course, I also review some tickets over the past two weeks:

#532 - Big jump from finger18.py to finger19.py in tutorial
#918 - If program has > 1024 fds, the select() used in t.i.process can fail (closed)
#1009 - Document defgen
#2110 - finger tutorial uses unclear python idioms (closed)
#2682 - twisted.conch.ssh.userauth is poorly tested
#3245 - __slots__ in Deferred class
#3498 - conch connection-sharing should be disabled by default, at least until it works a little
#3575 - Common implementation of RFC 2617 digest authentication (closed)
#3584 - SIP transaction layer
#3613 - a streaming producer doesn't have to be resumed when a buffer is empty (closed)

Several documentation tickets there, as well as some good SIP stuff (more of Allen's work). The digest authentication ticket (#3575) was a nice one. Twisted has something like three different implementations of digest authentication in it, each for a different protocol with slightly different requirements. #3575 is the beginning of the consolidation of that code, a pretty important task.

Meanwhile, Allen has been working on the SIP code in Twisted in parallel with my development. A report of his work will show up here shortly.

Thanks to the SFC and all of the sponsors who made this possible, as well as to all the other Twisted developers who helped out by writing or reviewing code.

Summary of December Sponsored Twisted Development

2008-12-30T06:45:00.000-08:00

With a lot of help from Allen Short, two more weeks of sponsored Twisted development have just been completed. During this round, I continued to work on the new HTTP client and spent some time trying to resolve some long-standing issues around Twisted's support for starting and controlling child processes. Meanwhile, Allen developed a plan for integrating the SIP code, maintained by Divmod, Inc. outside of Twisted for several years, into Twisted and the existing SIP support in Twisted.

The SIP-related work that Allen did encompassed these tickets:

#2194 - small bug in SIP Via header generation
#3575 - Common implementation of RFC 2617 digest authentication
#3582 - Improve SIP URI parsing/formatting
#3583 - Include SIP message-parsing changes from Sine
#3584 - SIP transport layer and transaction layer

These are the process tickets that I worked on:

#733 - twisted's SIGCHLD handler breaks popen
#1997 - perhaps wakeUp could be slightly simpler (closed)
#2967 - Reaping child processes has superlinear complexity on POSIX
#3571 - intermittent spawnProcess failure in test_process on Linux (closed)
#3576 - add high-level, cross-platform close-on-exit togglers to t.i.fdesc

A lot of this work consisted of getting some old code which Glyph had previously worked on but never completed updated and brought closer to being ready for a real code review. The summary of #1997 is a bit misleading - while the change did simplify the reactor's "wake up" mechanism, it also removed a race condition, fixing a bug which could cause certain events to fall into a limbo where they could only ever be processed after another event arrived.

Aside from process related tickets and the HTTP client, I worked on a few other tickets as well:

#2808 - AMP should raise MissingArgument (or other) if a callRemote is called with wrong arguments (closed)
#3246 - remove all mentions of plugins.tml from the documentation (closed)
#3562 - Setup Python 3.0 buildslave(s) (closed)
#3568 - ERROR from conch test when pycrypto is not installed
#3569 - Twisted Web WSGI container sometimes emits too many (or duplicate) headers (closed)

The Python 3.0 buildslave will probably garner the most interest of the bunch. The resolution of this ticket does not mean that Twisted supports Python 3.0 now. It just means that we've added a column to our buildbot (continuous integration system). We can now tell at any given time that Twisted does not support Python 3.0. ;) But seriously, with this slave set up, we can accept contributions which move Twisted towards Python 3.0 compatibility, but I don't plan to spend any time doing such development myself in the near future. There's a ton of other more pressing issues, so I'll leave Python 3.0 work to people who think they'll benefit from it.

As for the new HTTP client, this round of development moves it inexorably towards completion. Itamar Shtull-Trauring and I spent several days improving its error handling, simplifying some of the more hideous parts of the implementation, and dealing with various corner cases (and HTTP 1.1 sure has a lot of them). The development branch also includes a sketch (only a sketch) of the higher-level API we're planning to provide on top of the low-level protocol implementation (the sketch is currently undocumented and a bit obtuse, so it may not make sense without me or Itamar looking over your shoulder) and an example which uses the APIs provided by the low-level protocol to implement a simple web client (something along the lines of wget or curl, but obviously much, much, much more rudamentary).

That's it for now. It'll be 2009 when I post the next one of these. 2008 has been a great year for Twisted development, and I know that things are just going to get better. :) Thanks again to the Software Freedom Conservancy, all of the Twisted Sponsors, and all the other developers who contribute to Twisted.

Summary of November Sponsored Twisted Development

2008-12-03T05:44:00.000-08:00

I've just completed another round of sponsored Twisted development (the sixth so far). This period sees a lot of documentation improvements, as well as various bug fixes and continued work on a new HTTP client for Twisted Web (#886).

These are the documentation issues which were resolved:

#2281 - Annotations for Twisted Finger Tutorial
#3548 - twisted.conch.client.knownhosts.PlainEntry misdocuments its "_hostnames" attribute.
#3455 - CONNECTION_LOST not an Integer...
#3537 - Conch's public key authentication process is confusing.
#3490 - FTPClient errors should provide ftp errorcode

There were some improvements to Twisted Web:

#1878 - twisted.web.monitor traceback_ AttributeError: class IChangeNotified has no attribute '__class__'
#2402 - client.py crashes on URL's that would be no problem for most browsers
#3192 - HTTPClientFactory sets followRedirect on the HTTPPageGetter class
#3469 - Exception is rendered when NotFound is more appropriate.

A problem with Conch's SFTP server's reporting of modification times was fixed (and then a problem with the unit tests for the fix was fixed):

#3503 - Wrong date format delivered by twisted.conch.ls.lsLine
#3551 - TZ=Pacific/Auckland python2.4 ./bin/trial twisted.conch.test.test_cftp.ListingTests fails

With #3551, I was once again reminded that working with timezones is extremely challenging. This time, I found that the range of local renderings of any particular UTC timestamp is greater than 24 hours, so you cannot rely on any particular timestamp falling on a particular day in an arbitrary timezone. So like other unit tests in Twisted, the unit tests for lsLine explicitly set the timezone when they want to make assertions which depend on it, then reset it to its original value. Unfortunately, this means they can't run on Windows, since Windows apparently lacks APIs for doing this. I also rediscovered that time.tzset() is a no-op, at least on Linux (but the unit tests call it anyway, in case Solaris or HP-UX or some other POSIX implementation requires it).

An important issue with Twisted's Jabber support was fixed as well (#3463). This one prevented Twisted's Jabber client from successfully negotiating TLS when connected to Google Talk (and possibly other Java-based Jabber servers). This problem was ultimately caused by a bug in the TLS support on Google Talk which caused TLS negotiation to fail if the client included a session ticket (RFC 5077) section in the handshake. This is allowed and servers which do not support session tickets should ignore the section, but for some reason it causes problems with Google Talk. OpenSSL (on which Twisted's SSL support is based) 0.9.9 enables session tickets and the 0.9.8 package distributed with some platforms (eg Ubuntu 8.10) includes a backport of this feature. So Twisted's Jabber client cannot communicate with Google Talk if one of those versions of OpenSSL is installed.

A couple issues related to Python 2.6 support were fixed:

#2763 - md5 and sha module will be deprecated in python 2.6
#3545 - Support Python 2.6 in the Windows build system.

A bug in Twisted Mail's IMAP4 client which prevented the unseen part of a server's response to a select or examine command from being made available to applications was fixed (#3550).

And there were several other assorted fixes:

#3521 - Documentation for `processExited()` conflicts with the implementation
#3315 - t.p.reflect.safe_repr includes the wrong traceback and misformats the return value
#3541 - twisted.internet.abstract.FileDescriptor.loseConnection drops reason (will be reported as clean shutdown)
#3544 - bin/admin/change-versions should update the main README file

I also spent some time working on converting Twisted Lore from using microdom, Twisted's XML parser and DOM implementation (circa 2002), to using minidom, the XML parser and DOM implementation in the Python standard library. For a long time, microdom was better than any of the alternatives, but it's seen very little maintenance in the past several years and there are some problems with Lore (eg #414) which are caused by behavior of microdom that it would be difficult to change.

Unfortunately, switching to minidom brings a new set of problems. It's still probably worthwhile, but it seems like it's harder to use than it should be. Some of the issues I've run into so far (and I'm not done yet):

minidom's constructors are less convenient than microdom's constructors. For example, the Text constructor doesn't accept a string to use as the text node's value. Instead, you have to instantiate Text and then set an attribute. This expands code which should have been one line into three lines. And to make things worse, a Text instance with no data set raises an exception from the __repr__ method.
The parse error exceptions raised by expat are less informative than the parse error exceptions raised by microdom. For example, if a document contains mismatched tags, microdom reports the name and location of both the opening and closing tags. minidom will report only the location of the closing tag. It's easy enough to find out the name of the closing tag by finding that location in the input document, but finding the offending start tag means parsing the document in your head. For any non-trivial document this is ridiculous. Fortunately, by switching to sax and providing custom error and content handlers, most of the information can be recovered. This information is always useful though, and it would be better if minidom provided it by default.
Once you switch to xml.sax, you have to remember to disable its validation features or it will try to retrieve DTDs from the internet every time you parse something. This is bad, bad default behavior.

Some of these issues may turn into Python bug reports once I've made more progress on converting Lore. A much bigger difficulty with the conversion than the problems minidom has is the fact that Lore is largely pre-UQDS code. Some of it has tests, but they're mostly whole-system tests which compare gigantic xhtml strings and are subject to extremely obscure failures. And most of it doesn't have any tests at all.

Switching away from microdom should be worth the effort though. minidom is a bit faster and Lore will generate better looking output once the switch is done.

Itamar points out I didn't separate tickets into groups for those I reviewed and those I did development on myself this time. So let me point out that of the above tickets, many were developed by others and reviewed by me. The Twisted development process is highly collaborative. I couldn't accomplish anything without the help of all the other great Twisted developers who volunteer to contribute to Twisted in their free time. If you want to find out which are which, head over to the Twisted issue tracker where you can look up the development(/authorship/etc) history of any ticket.

That's all for now. Thanks again to the Software Freedom Conservancy, all of the Twisted Sponsors, and all the other developers who contribute to Twisted (hi Michael!).

Summary of October Sponsored Twisted Development

2008-11-03T15:31:00.000-08:00

Hello again,

I've just spent another two weeks on sponsored Twisted development. This round sees 25 tickets resolved. Considering the fact that two tickets in particular took up a lot of development time, I'm pretty happy with this progress. I did a lot of reviews for other people this time around. Here are some of the more interesting things developed by other people to which I gave the thumbs up:

#2026 - Edit the twisted.internet docstring to make contents clearer
#2902 - bad flag value in twisted.conch.ssh.filetransfer
#3335 - FTPClient should support renaming files
#3491 - FTPClient should support deleting files
#3500 - Add directory creation method to FTPClient

(The other tickets I reviewed which weren't quite as as interesting were: #532 #745 #1853 #2281 #2375 #2682 #2748 #3197 #3216 #3315 #3381 #3393 #3439 #3464 #3466 #3471 #3482 #3486 #3504 #3505 #3507)

And here are some of the more interesting ones I worked on myself (bold tickets are now closed):

#886 - Rewrite twisted.web.client.getPage to support streaming of result data and returning headers etc
#2346 - Header generation not conforming to RFC822 and RFC2046
#2605 - stdlib unittest change breaks a trial test
#3298 - twisted.internet.defer.FirstError masks errors when logged
#3329 - HTTP's #(...) syntax allows null contents
#3477 - BinaryBoxProtocol.makeConnection calls startReceivingBoxes too early
#3478 - AMP should enforce MAX_KEY_LENGTH in the protocol parser
#3487 - Add TestCase.flushWarnings

(The other slightly less exciting tickets I worked were: #3169 #3197 #3216 #3515 #3519)

#3487 and #886 took up a lot of my time this round. With #3487 resolved, the way is clear for the next Twisted release to run well on Python 2.6 (ie, there are no more known issues). Once I got that out of the way, I moved back to web issues. #886 is going to represent a big step towards having a good HTTP client API in Twisted. Anyone who has tried to use twisted.web.client for anything non-trivial knows how much work is needed in this area. :)

As usual, this work is made possible by the SFC and all of the sponsors who made this possible, as well as to all the other Twisted developers who helped out by writing or reviewing code!

That's all until next time.

Summary of September Sponsored Twisted Development

2008-09-26T12:28:00.000-07:00

Another round (the fourth) of sponsored Twisted development has just wrapped up.

I had a lot of excellent help again from Thomas Hervé and Thijs Triemstra. Thomas is a great reviewer and Thijs has been making steady progress on resolving lots of documentation tickets.

Here are the tickets I spent time developing:

#1068 - UNIX Port creation has race condition in permission setting code
#1382 - twisted.web.client.HTTPClientFactory sends Host header (and others) more than once
#1903 - Twisted doesn't reset timeout when getting POST data
#2542 - twisted.web.microdom.Node.isEqualToNode does no actual comparison
#2683 - twisted/web/http.py has self._header instead of self.__header
#2878 - Intermittent unclean errors from twisted.test.test_ftp.FTPServerPasvDataConnectionTestCase.testTwoDirLIST on OS X
#2947 - intermittent twisted.test.test_pb.NSPTestCase.test_NSP failure on OS X
#2974 - twisted.vfs.test.test_vfs.OSVFSTest.setUp does not close the files it opens
#2976 - twisted.mail.smtp.xtext_encode and xtext_decode take the wrong number of arguments
#3133 - Whois function for the ircclient class
#3211 - twisted.python.log.showwarning doesn't take the new line argument added in Python 2.6
#3222 - Intermittent failure of twisted.test.test_twistd.AppProfilingTestCase.test_profileSaveStats on Windows
#3223 - twisted.trial.unittest.TestCase.assertWarns always fails when the C warnings module_ new in Python trunk_ is in use
#3239 - Intermittent failure of twisted.test.test_iutils.UtilsTestCase.testOutputWithErrorIgnored on Windows
#3266 - Provide tools for managing new deprecation policy
#3402 - Intermittent failure of twisted.test.test_tcp.LocalRemoteAddressTestCase.test_hostAddress on OS X
#3404 - twisted.test.test_process.ProcessTestCase.testManyProcesses fails on WinXP
#3416 - rename `twisted.web2.iweb.IOldRequest` to `twisted.web.iweb.IRequest`
#3424 - Intermittent failure of twisted.names.test.test_names.ServerDNSTestCase.testZoneTransfer on OS X
#3426 - twisted/internet/test/test_process.py tests which might fail while waiting for an event should use runReactor instead of reactor.run

As you can see, work improving Twisted Web continues here. There's also been some long-needed fixes to tests in the suite which fail intermittently on various platforms. With more of these problems resolved, it gets easier and easier to determine if new proposed changes introduce any regressions.

Of course, I reviewed a lot of tickets over the last few weeks as well. Here they are:

#532 - Big jump from finger18.py to finger19.py in tutorial.
#1124 - getHost and getPeer not needed for IProcessTransport
#1157 - web.http.HTTPClient does not support chunked transfer-encoding
#1262 - Document Twisted copyright policy
#1328 - People keep asking about _trial_temp
#1852 - Replace uses of @ivar in interfaces with z.i.Attribute
#1853 - Fix/eliminate @cvar in Interfaces.
#2026 - Edit the twisted.internet docstring to make contents clearer
#2500 - Add @since to the coding standard
#2514 - Documentation bug in the list of possible classes
#2555 - Documentation update: deferred results in PB
#2909 - Document XXX Comment Policy
#3045 - Coding standard should forbid the use of "foo %s" % notATuple
#3197 - IProcessTransport.pid is incompletely documented
#3236 - Remove the FAQ from doc/core/howto_ since it is a duplicate of the FAQ wiki page
#3254 - twisted.python.deprecate does the wrong thing for methods
#3315 - t.p.reflect.safe_repr includes the wrong traceback and misformats the return value
#3382 - coding-standard.xhtml mislinks to twisted.python.compat
#3414 - Coding standard refers to old tools
#3415 - Get rid of X{} references in docstrings
#3422 - Remove acceptance tests in coding standard
#3423 - Get rid of references to old admin tools
#3435 - XmlStream should be convenient to use in a server context where there should be one authenticator per connection
#3446 - Infinite loop/memory-usage in irc.split()

These tickets represent a lot of good work by other Twisted developers (and you can see there are a lot of documentation tickets here, almost all the work of Thijs!).

I also spent a bit of time cleaning up our issue tracker. When you've been keeping track of bugs and feature requests for more than five years, it's hard to avoid having a certain amount of invalid tickets pile up. It's a bit of a drag going through the issue tracker to try to find these, but it's well worth while (and I can't expect everything to be fun ;).

That's it for this time. I'll be back with another report in about a month.

Summary of July/August TSF Sponsored Twisted Development

2008-08-05T06:54:00.000-07:00

I've just completed the third two-week period of TSF sponsored Twisted development.

Tickets I worked on during this period:

#686 - [TEST] startTLS is broken if there's already data in the outgoing buffer.
#966 - [PATCH] Add --umask option to twistd
#1200 - twisted.test.test_internet calls reactor.iterate()
#1493 - static File web module doesn't support byte ranges
#2276 - Changes to TwistedNames to make it support NAPTR records
#2338 - trial should handle concurrent usage in the same directory gracefully
#2753 - twisted.web WSGI support
#2790 - UDP Transport write() raises socket.error EWOULDBLOCK
#2845 - twisted.internet.thread._putResultInDeferred should be public
#2931 - FilePath.setContent writes and renames but does not sync
#3342 - twisted.names dns-spoofing vulnerability
#3347 - twisted.names dns-spoofing vulnerability (birthday paradox)
#3367 - twisted.python.lockfile.FilesystemLock.lock fails with EEXIST

Tickets I reviewed during this period:

#637 - Odd filenaming in tutorial/intro.xhtml
#638 - Allow overriding twistd's logging options
#689 - twistd man page needs a section on signals
#1246 - reactor.callWhenRunning is not in the Using Processes document
#1253 - Create index.xhtml files for non-core doc trees
#1490 - Allow twistd to "run" packages_ e.g. 'twistd run mypackage --port 8080'
#1821 - Turn deferredgenerator wiki page into howto
#1888 - Review man pages
#1971 - Links in SEE ALSO section of doc/lore/man/lore.1 are bogus
#2208 - Standardize on the Python shebang line
#2375 - these objects' docstrings are not proper epytext:
#2438 - Get rid of references to maintainer email addresses from code
#2607 - conch.checks.SSHPublicKeyDatabase calls os.seteuid/os.setegid even if it's not necessary
#3244 - runInteraction exceptions are swallowed if rollback fails
#3254 - twisted.python.deprecate does the wrong thing for methods
#3285 - Add ISUPPORT implementation for irc.py
#3332 - SSHAgent implementation_ unit tests
#3355 - t.application.app.AppProfiler handles options oddly
#3365 - a few more IRCClient docstrings
#3366 - add IRCClient.back()
#3377 - words.protocol - irc.py assumes nickchange to be successful before server ack

Allen Short also helped out with a few reviews this time. Thijs Triemstra, a new Twisted contributor, has also been very active lately working though documentation tickets and making great headway.

Twisted's open ticket count also fell below 1100 this week. For the last three months, for the first time ever, we've been able to consistently resolve more tickets than have been filed.

Thanks to the SFC (<http://conservancy.softwarefreedom.org/>) and all of the sponsors (<http://twistedmatrix.com/trac/wiki/TSF/FoundingSponsors>) who made this possible, as well as to all the other Twisted developers who helped out by writing or reviewing code.