Predictably Random

My Welcome Non-Committee

Fri, 06 Apr 2012 20:16:25 +1000

I’ve been home from PyCon US for three weeks, which has given me plenty of time to reflect on the experience. Now that the Easter break has started, I also have the time to put some of those thoughts into words.

PyCon US this year was huge, or so I was told. Many of the 2250+ attendees were, like me, making their first appearance. Maybe consequently, there was a strong theme of ‘welcome’ for the conference. Stormy Peters encouraged returning attendees to meet somebody new during her keynote address. People wore funny hats and called themselves the Welcome Un-Committee. And so on.

As great as these initiatives were, what made a strong difference for me was the group of people who consciously went out of there way to welcome me to PyCon. Sort of my own, personal, Welcome Non-Committee. What I want to do now is publicly acknowledge and thank them for this. It’s quite a long list of people, so after a couple of weeks of soul searching I created a list of five names that have made a lasting impression.

David, without doubt one of the friendliest, engaging persons I have ever met. One day I hope to visit Canada and be able to see one of the modular, futuristic, signage displays you work on. Thank you.

Maciej, made joining PyPy for sprints the welcoming, entertaining and over all fun experience it was. The enthusiasm you imparted on me leaves me feel guilty for not working on PyPy even while I write this. Thank you.

Taavi, even after giving his highly rated presentation to PyCon was happy to have a conversation with me on the topic at dinner. I still smile when recalling the steady stream of often hilarious 'WTF’ on NumPy edge cases you delivered at the sprints. Thank you.

Jonas, kept me company during the final hours of sprinting and shared some beers supplied by my travel allowance. I enjoyed our gentle jibing over whose fault it is that pymaging runs slower on PyPy.

Andrew, a fellow Australian, government employee, PyCon first time attendee. Being able to share the roller coaster that is PyCon tutorials, conference and sprints while spending a fortnight away from our families made an enormous difference. Thank you.

Of course, there is a very long other names that I could/should have included.

Simon, I really enjoyed discussing all things radio with you.
Dan, I’m amazed you chose to share a drink with me after the sprints wrapped up.
Michael, o.m.g, I met the guy behind mock and unittest!
Tavis, I was relieved to find someone who’d attempted a presentation that’s a little unusual, although dictating code is so much cooler …
James, o.m.g, I met the Eldarion guy!
Yarko, such positive enthusiasm is rare and inspirational. That and packing the swag bag was a blast.

As Steve Holden said, “people come for the programming language and stay for the community”. Each of you, as well as all the persons I was unable to name above, are that community for me. Thank you. I hope we meet again.

This is one sad panda. NERF and other projectile weapons (toy...

Thu, 22 Sep 2011 21:11:49 +1000

This is one sad panda.

NERF and other projectile weapons (toy or otherwise) have been banned in the office. I guess this is the final cease fire for The Great Office War.

Viva Le Resistance!

Missing the basics

Tue, 20 Sep 2011 22:45:16 +1000

If asked what you would miss from your modern life style, how would you answer? The internet? Your mobile phone? Perhaps electricity? Would you have considered answering with running water?

Yesterday we arrived home from a long weekend away to discover a leaking pipe. Fortunately it was a small leak and (we believe) fairly recent. Unfortunately it was inside the brick exterior wall. As we discovered it at the end of the day, we had to wait till the next day for a plumber to come and repair it. And so the mains tap was turned off.

In the hours that followed turning off the main water supply I realised just how much I take running water for granted. Simple practices such as washing hands and cleaning teeth became vastly more inconvenient. I was grateful to have done a load of washing almost as soon as we got home.

The pipe is repaired now, the baby is bathed and life has quickly returned to a comfortable existence. Hopefully in the future though I will better appreciate what those less fortunate than us don’t have with regard to clean, running and drinkable water.

Hidden Features

Mon, 19 Sep 2011 21:40:16 +1000

How do you wind down after a weekend whirl wind interstate visit? By perusing the Hidden features of Python thread on Stack Overflow of course!

Ok, so strictly speaking it wasn’t for winding down, more that the farm has incredibly limited wireless internet. That and I was unable to fit my laptop in amongst the other 47kg of luggage that comprises the family Iles travelling road show.

Seriously though, the thread has some impressive answers. I’m surprised it hasn’t made it onto Jesse Noller’s Good to Great Python Reads yet.

"Never mistake motion for action."

Wed, 14 Sep 2011 20:31:05 +1000

“Never mistake motion for action.”

- http://www.brainyquote.com/quotes/quotes/e/ernesthemi103727.html

Wikipedia book creator

Tue, 13 Sep 2011 20:31:05 +1000

I stumbled upon a most welcome new feature on Wikipedia the other night. The Wikipedia Book Creator.

Wikipedia now enables users to create their own books from a compilation of wikipedia articles. Books can be downloaded as PDFs or printed through Pedia Press. 10% of the sales of printed copies goes to the Wikimedia Foundation.

Here’s my first book; String Algorithms, a collection of string searching and matching algorithms. I strongly recommend checking out the Aho-Corasick method.

Where does heapq belong now?

Mon, 12 Sep 2011 20:32:05 +1000

Two of my favorite featues in Python’s standard library are namedtuple and heapq. Namedtuple resides in the collections module, while heapq is a module in its own right.

While heapq predates collections. The standard library has never been considered a fossilized structure. Reorganisation has (and will) take place when necessary.

So, is it time to relocate heapq to a logical home in collections?

"My grandfather once told me that there were two kinds of people: those who do the work and those who..."

Sat, 10 Sep 2011 20:30:06 +1000

“My grandfather once told me that there were two kinds of people: those who do the work and those who take the credit. He told me to try to be in the first group; there was much less competition.”

- http://www.brainyquote.com/quotes/quotes/i/indiragand136134.html

Why I do the things I do.

Fri, 09 Sep 2011 20:31:06 +1000

Why I do the things I do.

No convergence for Chrome

Thu, 08 Sep 2011 22:03:28 +1000

No convergence for Chrome:

Looks like Chrome wont be getting native Convergence support, which is not really surprising. More interesting though, Chrome could get an API for “controlling certificate decisions”. This would make it possible to add support for Convergence, Perspectives or maybe even bittorrent based peer-to-peer notarisation as a Chrome extension. It will be interesting to see if someone steps up and provides a patch to add this API.

Chrome, why dont you?

Wed, 07 Sep 2011 21:51:54 +1000

Following yesterday’s post on flaws in the trust model of SSL certificate authorities; today I had a conversation in the break out area with Indra on how browsers manage certificates. Indra reminded me how Phobos, one of the Tor developers, removes all certificate authorities from his/her Firefox setup. Phobos instead adds certificates individually to Firefox as he/she gains trust in a site.

That’s impressive, but extreme. The amount of effort, knowledge and technical skill required to achieve an outcome is beyond of most browser users.

My browser of choice is Google’s Chrome. Chrome already has some additional security features for SSL certificates and secured domains, such as certificate pinning. But Chrome doesn’t give me any tools to even partially emulate Phobo’s approach. Which left me to lament to Indra:

Why aren’t certificates for my important domains (e.g. internet banking) already pinned?
Why can’t I manually pin the certificate for select domains?
Why can’t I declare some (or all) SSL protected domains to be sensitive?
Why doesn’t Chrome remember the last certificate presented by sensitive domains?
Why doesn’t Chrom warn me when the certificate for a sensitive domain changes?

So Google Chrome, why don’t you?

Fixing SSL flaws

Tue, 06 Sep 2011 23:28:20 +1000

Just in case you missed it; a very serious security breach occurred recently at DigiNotar, a dutch certificate authority. At least 531 fraudulent certificates were issued by the attacker for a wide variety of SSL/TLS secured sites including Gmail, Facebook and the Tor project. Some of those certificates have been used in man-in-the-middle (MITM) attack on Iranian citizens.

One of the effects of this event, and the earlier Comodo breach, has been a renewed discussion on “fixing” SSL because it’s broken. Ignoring the argument of whether SSL is “broken” or just flawed, so far I seen two basic themes to the proposals.

The first is decentralise the trust relationship. Solutions like Perspectives and Convergence introduce the concept of “notaries”. Notaries monitor the use of certificates on sites over time. A user chooses which notary or notaries they trust; when accessing a SSL secured site, the notary is contacted to verify the server certificate presented to the browser.

Which makes we wonder why not do away with the notaries altogether. The recent implementation of uChat, a peer-to-peer chat application built on bittorrent. Using magnet links for each site, swarms could assemble to clients distributed around the world to compare the certificates being presented to each. All this could be built directly into the browser.

The second is to increase the cost of attacks against certificate authorities by have multiple independent certificate authorities sign a certificate. This would require an attacker to compromise multiple certificate authorities simultaneously to issue a fraudulent certificate.

Avoiding any discussion on the relative security of the two models for the moment. I contend it’s the second approach that will prevail. Why? Performance.

A lot of work has been going of recently to reduce the latency of creating new secured connections. Technologies like session resumption, FalseStart, SnapStart, and OSCP stapling are all designed to reduce the number of round trips to create a connection. Decentralising the trust relationship doesn’t just introduce many more rounds trips than were required in the past, it introduces round trips to multiple new locations.

Compounding the problem of a poor user experience, I also contend that a distributed trust model is more vulnerable to state level censorship. When the communication required to establish a new secured session is restricted to that between the client and the server (baring fraudulent certificates) a state level actor is contained to denial of service.

However, in a distributed model the state level actor has the possibility of allowing access to the service but denying access to any notaries or peers outside the censored region. Thus the client, and ultimately the user, is afforded the option of accessing the service but without any guarantees as to the security of the site. We all know how well users pay attention to warning messages today.

I believe this to be a significant weakness of the distributed trust model. A denial of service attack on the SSL validation infrastructure is potentially far more damaging than a denial of service attack on a service its self.

It will be interesting to watch dialogue in this space develop. There are a lot of talented security professionals I’ve yet to see comment such as Bruch Schneier and Adam Langley. Let’s hope as few people as possible get hurt until a solution is found.

Why metaclass?

Mon, 05 Sep 2011 23:12:58 +1000

I was asked an excellent question today:

So those metaclasses you’ve ben writing about, when would you use them?

I thought this question was great because, if you’ve been following my previous posts on using metaclasses, you may have noticed I never stopped to describe some scenarios when metaclasses are useful.

A good place to start is this 2002 quote from Tim Peters, the Tim behind timsort.

[Metaclasses] are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don’t (the people who actually need themknow with certainty that they need them,and don’t need an explanation about why).

While not exactly answering the question at hand, it’s a pertinent reminder that just because we have a hammer it doesn’t mean we need to treat every problem like a nail.

So when might you know you need them? Metaclasses are the class of classes, so they’re capable of making systematic changes to class definitions or class instances as they are created. When multiple classes need to adhere to common interface, that logic can be abstracted out into a metaclass.

That’s very abstract, so here’s a short list to real world examples:

A properties metaclass to create properties from getter and setter methods.
A selfless metaclass to automatically add ‘self’ to all method definitions.
A singleton metaclass to create singleton objects.
Self registering subclasses in a plugin system.
Declarative programming as used by Protocol Buffers and SQLAlchemy to dynamically generate class definitions.
And my namedtuple metaclass and constructor metaclass.

What all these examples have in common is the application of a common transform to class definition or instance creation. That transform could add new methods or properties, modify existing methods or properties, or communicate with an external framework.

Metaclasses are a specialised tool. While invaluable for framework authors, it’s rare to use them in standard application code. If in doubt, refer to the Zen of Python and ask yourself; 'Is using a metaclass making my code clearer, more concise and more maintainable?’.

Boo Ya, Cats

Fri, 02 Sep 2011 23:13:26 +1000

Tonight I had a plan. That plan was to clean up filemagic and release it on PyPi. That plan went out the window courtesy of the Geelong versus Collingwood tonight.

For those who don’t follow AFL; this is the final round before finals in the AFL. Collingwood and Geelong are the first and second placed teams respectively. Should be a epic clash, Collingwood are the favourites to win.

However, the final score stands by its self.

Perhaps I should limit myself to to a hearty ’Go Cats!’ … but I can’t. Please click through the Geelong emblem above for a little celebratory video.

QR code buddy icon

Thu, 01 Sep 2011 22:20:24 +1000

Every few months I get bored and change my profile picture on horde of sites I have accounts on. This time around I got geeky and used a Quick Response (QR) code.

QR codes have been around for nearly 20 years now. While popular in Japan they are only just beginning to become common here in Australia. I was inspired to use one as my profile picture when I found QREncoder on the Mac App Store. Although I confess I stole the idea from Mike Cardwell.

Generating QR codes is very easy. In addition to desktop applications there are multiple websites including Google charts that will generate charts for you. A Google search will give you an extensive list of options.

An interesting property of QR codes is the use of Reed-Soloman error correction to make scanning more reliable. This allows errors to be introduced to customise the design, make them more visually appealing or embed corporate logos. Clearly this will have some impact on how reliably readers will scan your code. (Thanks for Indra for passing me that last link.)

Now, if I can figure out how to drive Pixelmator, my next profile picture might look a little more spectacular.

Starting 'The List'

Sun, 28 Aug 2011 23:40:17 +1000

Over the past few weeks I’ve managed to start a number of interesting projects, but failed to bring many of them to a close. Or even to a stable state.

I’m not sure how others maintain focus. As a start I thought I’d write down the major pieces of work I’ve started recently.

main; I write a lot of small applications with limited use and longevity. This package is to provide convenience functionality to reduce the amount of Python boiler plate.
filemagic; exposes the magic of libmagic to Python.
jitterbug; a framework for profiling the behaviour of PyPy’s JIT compiler.

Of these ‘filemagic’ is the closest to complete. It’s mostly just lacking in decent tests, save a couple of minor features.

The 'main’ package has progressed the furthest and is the most usable. Before I could consider it stable I want to add some common application functionality such as unix daemons and logging along with default option sets for them.

Lastly is jitterbug. I am still most enthusiastic about understanding the PyPy JIT’s behaviour. But am finding it difficult to find the blocks of time I need to concentrate on the topic at length.

Now all I have to do is convince the boss there’s value in letting me progress some of these during work time.

The when of Python scoping

Fri, 26 Aug 2011 22:16:45 +1000

Here’s a fun bit of Python trivia I learned a yesterday. The decision to determine the scope of a Python variable is made at compile time not run time by the Python interpreter.

Compile time for Python you ask? Why yes. When Python code is first read by the interpreter it is translated into byte code that includes representations for the classes and functions in the code. It’s during this first read that Python determines a variables scope. As an aside, Python includes support for dissembling the bytecode which can be useful to performance analysis and debugging.

Normally this is not something you need to worry about. However, it is possible to introduce interesting bugs as a side effect of when scoping rules are applied. Consider this short piece of code:

from __future__ import print_function

def outer(name):
    def inner():
        name = name.capitalize()
        return 'Hello {0:s}'.format(name)
    return inner()
print(outer('world'))

This code will raise an UnboundLocalError for name. That’s because Python will determine when first reading the code that name is local to the inner function because there is an assignment to name. Even though name is available to to inner from outer’s scope.

Scoping rules are more than just interesting trivia. It goes to the heart of how closures work in Python. But that’s a post for another night.

This is my tomorrow.

Wed, 24 Aug 2011 22:10:40 +1000

This is my tomorrow.

Seriously, you can trust me!

Tue, 23 Aug 2011 22:10:07 +1000

Seriously, you can trust me!

Keeping up with Python

Mon, 22 Aug 2011 22:26:03 +1000

It’s hard keeping up to date with the “happenings” in an open source community as large and diverse as Python. I make no claims of knowing where the vanguard of Python news is. Though if it helps you, here’s how I stay up to date with developments in the community.

Most important to me is Twitter. The Python community is diverse, active and vibrant on Twitter. I’ve been acquiring new people to follow for well over a year now, so to help you get started I’ve started curating two Python related Twitter lists for you to follow:

The first list is of general members of the Python community I find interesting. The second list is of people predominately related to the PyPy project.

After Twitter I use Google Reader to follow Python related blogs and news feeds. To date I’ve acquired 47 Python related feeds. Again to make it easy for you I’ve collated these feeds into a single Google Reader bungle.

“Python” bundle created by Aaron

The bundle page includes an Atom link so you can follow along with your preferred news reader.

That should be enough to get you started following along with the Python community. If there are other interesting people and feeds you follow, please share them in the comments below.