For Some Value of "Magic"

A Statement from Adria Richards

2013-05-07T14:18:00.000-04:00

Having remained silent since PyCon, Adria Richards has now published a statement reflecting on those events, though sensibly without going into specifics. I think it's important that her retrospective view gets some air, so I am reproducing it here with my own comments.

Those who know me well in the the developer and tech community recognize that I have always tried to conduct myself in a way that builds bridges for everyone. My central aim is to do everything I can to help create new, inclusive inroads for all, no matter who they are, where they come from or what they believe. Development is about innovation, creativity, and in a grand sense, the betterment of human society through technology. So, it stands to reason that everyone should have a seat at the table, and everyone involved in this vital community should feel welcome, safe and respected. In essence, the worldwide community of developers can and should function as a reflection of what our wider society strives to be.

I don't know Adria, so I take her opening statement at face value. Those who disagree may do so elsewhere. Garann Means recently made a fairly good case that the denizens of the technology world don't act in a way that inspires respect from other communities. I think it's fair to say (and some of the comments in forums like Hacker News amply demonstrate) that there is plenty of room for improvement in what's considered acceptable conduct in the technology world.

I cannot comment at this time on the specifics of what occurred at PyCon on March 17, and the subsequent events of the following days, but I can offer some general thoughts. I don’t think anyone who was part of what happened at PyCon that day could possibly have imagined how this issue would have exploded into the public consciousness the way it has. I certainly did not, and now that the severest of consequences have manifested, all I wish to do is find the good in what has been one of the most challenging weeks of my life.

This positive attitude is to be admired. I have been through similarly shitty situations in my own life (as most of us have at one time or another) and while it's good to move on I like to extract lessons that will help me try to avoid making the same mistakes. If I can find those I can feel more positive about the experience.

And I do believe there is good to be found in this situation. Debate and recrimination can and must give way to dialog that explores the root causes of these issues in the tech industry. As developers and members of the startup community, we can welcome newcomers, women and people of color who, as of now, are under-represented in our ranks. And, all of us can learn a great deal from those who are well-established in the field. We can solidify the values of our workplaces (yes, conference spaces are workplaces!), and set new, positive and inclusive examples for other professional disciplines.

This is a good idea. PyCon was started to try and provide an inclusive space where everyone with interests in Python can come together for the betterment of all. I would be suspicious of those who don't support the above as an ideal. We surely have to accept that there's still along way to go to get there. So dialog is a good idea.

What happened at PyCon has cast a spotlight on a range of deep issues and problems in the developer world. As ugly as this situation has become, all of these issues have reasonable, and, I think, easily reached solutions that will help us cast conflict aside and construct a more cohesive and welcoming professional environment based on respect, trust and open communication. I do not, at this time, wish to concentrate on the fallout of the last several days. Instead, I want to be an integral part of a diverse, core group of individuals that comes together in a spirit of healing and openness to devise answers to the many questions that have arisen in the last week. Together, we can work to make the tech world a better place to work for everyone, and in doing so, we make the wider world a better place for all.

It's interesting that this episode happened at PyCon. When we started to talk seriously about Codes of Conduct in the Python world there was a lot of kick-back, and people said things like "but this is the Python community" (implying that we are such nice people that we won't have the same issues) and "but that's just normal behavior" (not even implying, but specifically stating, that anyone who needed a Code of Conduct to condition their behavior shouldn't be going to conferences anyway).

We pressed ahead none the less, refining the code to improve its expression of our ideals of openness and inclusivity. When issues had to be addressed (and you can read the reports in the PyCon blog if you need to refresh your memory) the code of conduct was applied, and to my mind it worked. The fact that two people lost their jobs as a result of the fall-out was nothing to do with the code of conduct, and yet there were many who felt that the blame should be laid at PyCon's door. I shudder to think what the result would have been at an event with no explicit code.

I hope that Adria can come back to PyCon without being subjected to any hostility. She expresses a great ideal, one that we should all be striving for. Now we just have to work out how to achieve it. But that's an implementation detail, right? I wish!

So Long, and Thanks for All the Fish

2013-04-29T01:34:00.001-04:00

A notable transition point in my life, by any measure. Interesting things are happening in all directions, many of them (I am delighted to day) heavily involving Python. So this is definitely not goodbye. My time on the Foundation's board has been a wonderful and interesting ride. I hope I have lived up to Tim O'Reilly's ethic of creating more value than I extract.

Why Tuples?

2013-04-28T00:06:00.001-04:00

A frequent question from new and even not-so-new Python programmers is "why does the language have both tuples (which, if you know Python, you will recall are immutable) and lists?" You might almost say there are two kinds of Python programmers, those who know what tuples are for and those whose mathematical education has been limited. I know this sounds like an awfully snobbish thing to say, but it isn't meant that way. The fact is that I learned most of my programming in eight years of working experience before I started my degree studies, and I am therefore of the very definite opinion that people with little mathematical background can be excellent programmers.

It's just that I myself only came across the word tuple when I started studying college-level mathematics and had to come to terms with things like

An NFA is represented formally by a 5-tuple, (Q, Σ, Δ, q₀, F), consisting of

a finite set of states Q

a finite set of input symbols Σ

a transition relation Δ : Q × Σ → P(Q).

an initial (or start) state q₀ ∈ Q

a set of states F distinguished as accepting (or final) states F ⊆ Q.

Now this has a very definite meaning. It tells us that an (ordered - i.e. positionally-identified) set of five things is sufficient to define the behavior of a specific type of object (in this case a non-deterministic finite state automaton (NFA), though for all you need to know about those I might just as well be describing a flux capacitor).

If we wanted to encapsulate an NFA as a Python data structure, then we might at some point in our code write in our Python code something like

nfa = (states, symbols, transition, initial, accepting_states)

though in actual fact you would be more likely to want to incorporate behavior in a class and so write instead

nfa = NFA(states, symbols, transition, initial, accepting_states)

Now even though you may not know what an NFA is, you will surely perceive that the set of its possible states is a very different thing from the function that determines how a current state and a set of inputs are mapped into new states. So there is simply no way that it would be meaningful to write

for thing in (states, symbols, transition, initial, accepting_states):
do_something_with(thing)

unless you were, for example, trying to save its state during a pickling operation or some more obscure book-keeping metacode.

And that, best beloved, is what tuples are for: they are ordered collections of objects, and each of the objects has, according to its position, a specific meaning (sometimes referred to as its semantics). If no behaviors are required then a tuple is "about the simplest thing that could work."

This is why you often hear people informally say tuples are for "collections of things you don't need to iterate over" or "tuples are for sequences of dissimilar objects". My advice would be to stay away from such discussions. One might reasonably argue that including them in the language discouraged people from using objects with named attributes, which are always easier to deal with.

The problem with the tuple is that once we have constructed our NFA the only way to refer to the states in code is with the rather unedifying expression

nfa[0]

which doesn't actually tell the reader much about the programmer's intention. The sequence nature of the tuple means that our accesses to the elements are difficult to imbue with meaning in our code. This latterly became such an obvious shortcoming that it prompted Raymond Hettinger to create the namedtuple object that allows you to easily specify a tuple whose elements can also be referred to by name.

It would be interesting to see whether users of namedtuple objects actually use them as tuples at all. I would guess that the sequence behaviors are rarely used, in which case perhaps it's time to either remove namedtuple's __getitem__() and similar methods or implement a similar object without the sequence behaviors.

Gun Nuts 1

2012-12-17T02:44:00.003-05:00

In the wake of the most recent school slaying I thought it might be relevant to revisit the writings of Eric S Raymond. Within my world he is a well-known and self-described "gun nut." His two principal writings on he right to bear arms appear to be The Parable of the Sheep and Ethics Through the Barrel of a Gun. I will look at the latter in a separate post. There is nothing I can do for the misery of the bereaved families. I can only hope to help in the longer run.

The Parable of the Sheep

This is a tale of a flock of sheep. The dog cannot keep the wolves away. Some sheep take the claws and fangs from dead wolves. They fight back, reducing the carnage. Eventually the wolves mostly leave sheep alone. They do not know which sheep are armed. Many sheep are terrified of the weapons. They ban the armed sheep from to the pasture. They post signs at the edges forbidding hidden weapons. The wolves, seeing this, return. Once again they inflict carnage on the flock. They are only beaten off by the armed sheep. The flock remain afraid of the weapons. They barely tolerate their protectors, the armed sheep. The armed sheep altruistically continue to protect the flock. Raymond describes the situation thus:

The bold sheep knew that the fangs and claws they possessed had not changed them. They still grazed like other sheep, and raised their lambs in the spring, and greeted their friend the dog as he walked among them. But they could not quell the terror of the flock, which rose in them like some ancient dark smoky spirit and could not be damped by reason, nor dispelled by the light of day.

The parable omits many refinements of the true situation. That is forgivable for expository purposes. A school shooting represents something outside the parable. There is a genuine reason for the terror of the unarmed sheep. Occasionally the armed sheep start acting like wolves. Of course the unarmed sheep are then the inevitable target.

The statistics for murder and armed robbery are quite alarming. Those for child murder are truly chilling. Almost 63% involved a firearm in 2011. Well over a thousand kids blown away each year. To suggest that we all arm to protect ourselves seems unbalanced. It's like the 1950's "defense" policy of Mutually Assured Destruction. Never was an acronym more appropriate. The terrorists aren't over there, they are among us. A school massacre is an act of terror.

Here the Japanese strategy of many approaches might be fruitful. It is unlikely there is a single switch we can turn to stop this happening. Surely it is better to try to understand these acts of atrocity. Consider them a societal ill and seek its cure. Perhaps we could try to build a world where people are better in touch with their fellows. Extremes of behavior can (most of the time) be recognized. If someone is heading in a dangerous direction, perhaps it can be deflected by sympathetic attention.

I imagine* that most mass murderers are completely in control most of the time. They are indistinguishable from "normal" people. The classic quote after an apparently psychotic shooting episode? "He seemed like a normal guy." Let's leave the fact it's usually a man to another time. It's not like they are landmines, liable to go off at any touch. I am a pretty placid person most of the time. Occasionally some perfect storm of circumstances hits. Shortage of sleep is often involved. Physical exhaustion never helps. Emotional distress is very rarely positive. I "go off on one" in an uncharacteristic way. We all have extremes of behavior, often in response to extreme circumstances. Some people are more extreme than others.

Some people, sadly, are so extreme that they must be incarcerated to protect the rest of us. I would, in the USA, prefer it if the government invested its funding in long-term plans to make the country a better place to live than lock up predominantly black predominantly drug-offending adults.** Now private companies are running prisons, vested interests are powerful.

So while Raymond's simplified argument seems reasonable enough, that's only because it is simplified. There are inevitable outliers in a flock of over three hundred million. Mental stability and degree of moral control are two important dimensions. At that scale the sheep would appear to have a good reason for caution. Anyone proposing to use fangs and claws should surely be assessed as a responsible individual. Mistakes will still be made, but (presumably) it is not beyond the wit of sheep to concoct a scheme that reduces the number of massacre incidents.

In network parlance, the parable doesn't scale to larger flocks. Neither does it scale to bigger and better teeth and claws. The reductio ad absurdum of many gun nut arguments is that we should all carry personal tactical nuclear weapons. What's reasonable, and what's not? Why are such deadly automatic weapons allowed outside gun clubs and the armed forces? I don't mind people enjoying handling and using firearms. Hunting, preferably for food, for example, is a perfectly legitimate use. So is fun at a gun club shooting range. Mowing down kids, not so much. We have to strike a balance.

One final note: please let's not imagine there is an instant cure. All we can do is lay, and maintain to, long term plans. These should aim to reduce the availability and access to weapons, particularly those capable of facilitating mass murder.

* Admittedly if this were a Wikipedia article someone would have at least added a "citation needed" flag
** Who are then unable to vote

A Thank You to Eric Sterling

2012-12-17T00:34:00.000-05:00

Offered for the attention of those who attended Eric's recent keynote at DjangoCon and, most especially for Eric himself. I still remember the talk and the panel session he took the time to contribute to fondly, and I particularly enjoyed the unique flavor he added to the lunch he attended.

I'm Sorry

2012-12-15T09:28:00.001-05:00

The one-eyed snake

It appears I've been that guy (again). I've seen those "sorry if you were offended" apologies, and they are bullshit. So let me begin by unreservedly apologizing for my inappropriate behavior. Then perhaps I can tell you what this is all about. I do so in the doleful expectation that this post will generate more heat than light, as is typical for this subject matter in geek circles, and will be the cause of a large amount of ill-informed, unhelpful and possibly personally damaging commentary. Since it gives me the opportunity to discuss a few issues of importance I accept that as the price.

~~Last~~ This year, at PyCon, I had a Square, a device to accept credit card payments through my cellphone, and inter alia was soliciting donations to the Python Software Foundation. As a part of my schtick I was offering to take photographs, and offered the option of a prop, a little Beanie Babies™ bean-stuffed Python which my dog, when a puppy, had once got hold of and played with so enthusiastically that she tore off (and probably swallowed) the glued-on bead that represented its left eye. I've had this snake almost ever since I started using Python (almost twenty years now) and I'm very fond of it. It often travels with me as a talisman, since I usually travel alone.

Because I am so familiar with it, in retrospect it seems a little odd that I had never particularly dwelt on the fact that it is, literally, my "one-eyed snake" though I had used the phrase at home with family and friends as a joke. [For readers not familiar with colloquial English and American, I am obliged to point out at this stage that "one-eyed snake" is one of many euphemisms for the penis]. I was amused enough, on that day, by this fatuous coincidence that I offered to include "the chairman's one-eyed snake" in the photographs I was taking, of men as well as women.

People appeared (in my opinion, not actually the opinion that matters as it happens) to tolerate my risqué attempt at humor, feeble as it was, and took it in good part although certainly not everyone asked for the snake to be included. The conference ran its course, I went on my way, unaware that a storm was brewing.

The storm's gestation in fact took another event to trigger it, a recent motion by the PSF board that we would require any conference we funded to implement a code of conduct. The board collectively felt that this would be a forward step, and would actually be helpful to conference organizers who had not yet addressed this issue. PyCon has has a code of conduct for several years, and since the conferences I run professionally also have codes of conduct, I had no hesitation in voting for the motion.

It's difficult to describe this situation in anonymous terms, since verbatim quotes are required in order to allow me to properly discuss the way this situation has turned out. But those who wish to know the identities of the proponents must do so by using {{search_engine_of_choice}}. What happened when news of the motion came out is that someone posted a comment on Hacker News which included the following content

From what I understand, the code was approved by the Board of Directors of the PSF, and not the PSF as a whole. Please, correct me if I am wrong. This is ironic since one of the Board members was walking around the conference last year with a damaged stuffed python toy asking, "Would you like to see my one eyed snake?"

This was said to one of my female colleagues. I asked her if she would like me to say something and she replied, "No, it is just creepy, but I'm an adult."

I'd like you to particularly note two things: first, the quote is inaccurate, I believe (if I uttered those words I would have to accept I would have gone over the line with anyone but a close friend); second, the colleague herself chose not to complain. This is significant because of what happened next. Clearly the comment had received some attention, since shortly thereafter a second person, one I think it is fair to say is known not to be well-disposed towards the PSF, tweeted (in reply to a third party who may for all I know have been a fifth or twelfth party)

Wait, I didn't read this comment closely enough. What fucktard was walking around asking women to pet his one-eyed snake?

Note again: the already inaccurate quotation has been through the mangler once more. So now we (that is me and the PSF, though in this post I must put my director's hat aside and accept personal culpability) have a situation. This is not something that would have worried me hugely in my personal capacity, but the fact is that it has brought a lot of pressure to bear on someone who, through his heroic efforts to improve on last year's incredible PyCon, is already under quite enough pressure. So it is necessary for me to write this explanation lest his efforts to bring in sponsorship and promote diversity of all kinds in the Python community are damaged by my bad behavior.

Now to the several points I should like to make about the outcomes so far. I hesitate to think of the fallout to my professional life.

I am lucky to be a man, since any woman posting about similar issues of female sensitivity is likely to be harangued and harassed until, unless she is of extraordinary character, she either allows herself to be silenced or leaves the Internet altogether (yes, well-known women have received threats that they felt obliged them to do this).
Context is important. Although my behavior may not have been appropriate, it was taking place in broad daylight in a well-populated area so nobody felt especially threatened by it. The same offer (let alone either of the two misquoted ones above) made to a woman who is the only other occupant of an elevator at 1:30 in the morning would, I think, be reasonably interpreted as a possible if not an actual threat. Anyone who doesn't understand that isn't really fit to be out alone.
It isn't up to me to decide whether my behavior is appropriate, but those with whom I interact. I teach professionally, have taken (and passed) sensitivity training classes, and am accustomed to behave in a professional manner that does not run the risk of attaching stigma to those I represent. Because I feel that many members of the Python community are my friends, I perhaps overstepped the boundaries of professionalism, believing that nobody would mind.
The fall-out of righteous indignation about events like this often lands on the wrong person. There is now heavy pressure on Jesse Noller as PyCon chair for not taking action about this. There have been definite suggestions that Jesse is being hypocritical in promoting a code of conduct for PyCon while condoning inappropriate behavior by a fellow director. There are two things worthy of note about this.
Firstly, since the issue only came to his and my attention eight months after the conference (and, indeed, after both he and I voted for the motion requiring codes of conduct) it is difficult to know what he could have been expected to do at the time. He has been heroically silent (for him).
Secondly, the code of conduct exists precisely to reassure people that harassing actions can be reported, and will be dealt with, no matter what the position of the offender in the community. So I think anyone who is gunning for Jesse about this should give him a pass. If he had come to me because he had received a complaint I cannot say what action I would have felt correct, but at the least I would have expected to make a public apology. I do not expect that Jesse would have felt able to let a transgression on my part go without action: he has far too much integrity for that. I like to think I have enough integrity to accept being told when I have transgressed (hence my dislike of the "if anyone was offended, sorry" stuff).
Circumstances alter cases. The code of conduct is not expected to protect people in a vacuum. If you find someone's behavior offensive your first recourse should be to say so. Hopefully an apology should be forthcoming, or at least a civilized discussion about your differences in standards. If the issue cannot be resolved in person then is the time to invoke the code of conduct. This is what makes variations in behavioral standards acceptable: when you are with people you know you are unlikely to go wrong; when you are not you should take others' input as defining acceptable limits.
It is unreasonable to expect perfection. We all slip from time to time, and a slip should not necessarily lead to a fall. I noted with a parenthetical "again" in the opening paragraph that this is not the first time I have behaved inappropriately, and sadly it may not be the last. I can say, though, that on the few occasions my behavior towards an individual was definitely capable of causing offense I have apologized sincerely and without reservation, usually before further contact from the other party (i.e. I am capable of recognizing bad behavior on my own part and voluntarily apologizing for it). I am happy to say that I count some of those to whom I have had to apologize as friends even now, and believe those feelings are reciprocated.
I specifically do not want the support of men weighing in with some equivalent of "what the hell, man, you didn't do anything wrong, what are the women complaining about?" Perhaps when your knuckles stop dragging along the ground we can talk. I very much would like the support of men who are equally liable to do stupid things (and have done) and yet still remain open to the possibility that one can be equally civil and courteous to members of all sexes. It's easy for men to say "we've all been that guy," but I hope that women don't imagine that guy conversation is laced with tales about how we were "that guy". We don't talk about our goofs even among ourselves (unless there's some club I'm not a member of). Men who are sensitive to the need for gender diversity find these mistakes painful to remember, and only dwell on them in private (this has not been an easy post to write). Men who are not sensitive to that need, please re-read the opening sentence of this paragraph. Support from others will also be more than welcome.
Diversity is not just gender diversity. Because I speak fluent English it is easy for Americans to imagine that I have absorbed their culture fully, though even after fifteen years I often have to stop to clarify cultural referents common to most natives. They forget that I am, in fact, an immigrant who was brought up with different standards. I am not trying to suggest that I should not be subject to the values of the USA, but would hope that when I overstep the bounds some leniency might be shown. I in my turn sometimes forget that standards are different over here.

There is more that I could write*, but I feel that this post is already long enough. I will close by remarking that it turns out the female colleague of the original poster is personally known to me, and in our several meetings both personal and professional since then she has not mentioned this issue (unless my memory has failed me). Naturally I intend to apologize to her at the first available opportunity, and to ensure that in the future I do nothing to make her feel uncomfortable (the basis on which I believed I was conducting the relationship). If I was not confident that people would feel comfortable in my presence I would not feel able to attend PyCon, since that would exclude others.

As PyCon's founder I genuinely want to see it address diversity issues properly. For eight years as a director and three years as chairman I have worked to ensure that the PSF is seen to take these issues seriously. I am mortified that I ran the risk of bringing either into disrepute. I hope that by making this apology I continue to lead by example.

* I might, say, discuss the appropriateness of using the word "fucktard" in public discourse on such serious matters .

Python for Data Analysis

2012-11-19T02:12:00.000-05:00

Python for Data Analysis; Wes McKinney. O'Reilly Media, October 2012
[Review copy kindly provided gratis by O'Reilly Media]

This book is a welcome addition to the Python canon, and anyone who is at all concerned with data analysis would do well to read it. From the opening chapter it is a clear and concise explanation of how to deploy Python for such tasks. Early on the author makes the point that Python should be thought of primarily as a “glue” language, so the book makes no attempt to show you how to program analytical methods in Python but rather demonstrates through practical examples how to use the existing extremely effective tool chain already available.

After a few brief motivating examples you will read one of the clearest expositions of the benefits of iPython I have seen. Since reading this chapter I have become convinced of the benefits of iPython Notebook and have opened an account with NotebookCloud*, which I feel will be a real boon in my teaching work.

The next chapter covers Numpy basics, followed by an introduction to the pandas library. You are then taken in detail, and with copious practical examples, through data cleanup and various important transformations, plotting and visualization, data aggregation and grouping, time series and finally financial applications.

The final chapter on advanced Numpy provides the kind of look under the hood that will help those interested to extract maximum efficiency from the package and understanding how best to take advantage of more advanced features. The closing Appendix, “Python Language Essentials,” is a concise introduction to the language that should provide even inexperienced Python users with sufficient information to understand the examples.

The writing is clear throughout, with numerous examples that enliven and illuminate the discussions. For those whose mantra is "show me the code," the code is here in spades. Flipping through the book you find very few pages that do not contain Python code (and many of them are displaying the output from Python code).

Wes McKinney took the sensible decision to focus on Python 2.7, which is what is currently deployed by the vast majority of the scientific and analytical communities. The quality of the writing and the recent welcome news that matplotlib** is now ported to Python 3 will, I am sure, make a Python 3 update a popular choice in coming years.

If you want to understand how you can use Python as an analytical tool then I heartily recommend this book.

* An Amazon Web Services account is required to use this service, and will be charged for your computations
** This means the standard "Scientific tool chain" is now available for Python 3

Building New Communities

2012-11-01T11:21:00.000-04:00

[This post is based on a keynote talk to the PyCon DE conference on November 1, 2012]

[UPDATE: Nov 17, 2012: The video of this talk is at http://pyvideo.org/video/1479/building-new-communities]

Good afternoon, and thank you for inviting me. It’s always intimidating to talk to an audience which is so technically capable. Today, though, I am hoping that we can look at a non-technical topic that has implications for everyone who is interested in Python: how do we promote the Python language and improve the development process?

DO EXISTING COMMUNITIES WORK?

Let me begin by saying that I have invested huge amounts of time in the Python community, and I believe that it has a well-deserved reputation as one of the more welcoming open source communities. So today I am talking more broadly than just Python. What I say applies, I believe, to many open source communities.

This talk is being given at an unprecedented time in human history. Clearly these things are a matter of interpretation, but nowadays it’s hard to find someone that doesn’t agree that the modern world is failing to meet the needs of many of its inhabitants.

Historically, we are living in strange times, The world human population reached 7 billion, depending on which figures you choose to believe, between October 2011 and March 2012, only twelve years after it reached 6 billion. According to one UN estimate it’s possible that population will reach ten billion before reaching any kind of stability, and that population in Africa could triple in this century. However you interpret these figures it is obvious that most governments are going to be struggling with population growth in a world where they are already finding it difficult to manage things to everyone’s satisfaction.

So I have a lot of questions, and not many answers. Being a geek, my approach attempts rationality—we have to take into account what we know about human behavior, differences between the various national and political cultures in which people live, and so on. In an increasingly crowded world it is easy to feel threatened by governmental and economic systems that are posited on continuous growth.

When I was young an organization called the Club of Rome produced a report called “The Limits to Growth,” positing that the finite nature of many resources could, if growth were not checked, lead to cataclysmic failures in supply of the very things that keep us alive: clean air, clean water, and nutritious food. This is where climate change deniers got their start, and yet from recent work it appears that the report was, within its limitations, a very accurate projection of certain futures.

My own vision of the future tends, I must admit, towards the catstrophic. I think it is entirely possible that the human race will wipe itself out simply by continuing to foul its own nest and ignoring the toxic environment it is creating. So I see it as important to try and bring this to people’s attention, and to do what I can to ensure that we (the human race) build a sustainable future for ourselves and the other species we have to share this huge ball of mud we call Earth with.

HEY, ISN'T THIS PYCON?

All of this might seem a very long way from our little open source world. I can already hear people thinking that I am abusing my position as a keynote speaker to stand on a political soap box and peddle my socialistic ideas. I admit I am a socialist, and those who know me also know that I am happy to stand on a soap box at times. But this is not one of those times.

Context, however, is everything. So I want to place my remarks today in the context of a world that isn’t working, that isn’t equitable, and in which ever larger divisions are opening up: between the poor and the rich in almost any country you like to name, between poor countries and rich countries, and so on. Huge amounts of resource are being wasted on machinery to wipe each other out. This is particularly noticeable in my current home country, the USA, which has 11% of the world’s population and yet is responsible for over 40% of all military expenditures, more than the next 14 top-spending countries put together. One might wonder what they are afraid of, but these expenditures certainly give them clout in the geopolitical sphere.

Open source communities, the Python community included, have growth problems of their own. According to Armit Deshpande and Dirk Riehle the number of lines of open source code is growing exponentially. This is a management problem of some magnitude, which is not being addressed at the moment. They suggest that open source applications generated revenues of $1.8 billion in 2006, and certainly growth has continued unimpeded since. Of course that $1.8bn represented only about three-quarters of one percent of total software revenues, so there is still a long way to go.

Tim O'Reilly made a sensible case this year that open source is responsible for increasing small business revenues of a single hosting company’s user base by $180 billion And, of course, many proprietary software products nowadays contain significant open source components.

GET PAID TO BUILD STUFF?

I have said many times that the emergence of open source represents a new economic phenomenon. It’s as though, when railways were needed in the United States, a bunch of people had said “You know, that’s a great idea. Let’s go build a railway system,” taken up their picks and shovels and headed off to work. Of course you can’t build a railway without access to heavy engineering, which is arguably difficult to open source. Similarly, most individuals would not find it cost-effective to make their own computers from scratch, or to interconnect them in the way that the infrastructure of the Internet does.

Just the same, the emergence of Linux as a powerful force in the software world, and the somewhat anarchic, less-than formal management of the whole Internet (despite the belief of many US politicians that the Internet should be managed by the US as a divine right) should convince us that it’s possible for individual and group initiatives to take hold and prosper in today’s rather strained commercial environments.

I understand that open source is not the exclusive domain of unpaid experts. Nowadays a lot of companies are paying programmers and other staff to assist with the creation of open source software projects. Many different models are used to try and recoup the investments made, but the fact remains: it is now commonly accepted that there is value to creating common infrastructure whose presence benefits a large proportion of the community rather than a single individual or organization. Of course such ideas often seem, particularly in the USA, socialistic and therefore suspect, but it is still true that large companies like HP, IBM and Microsoft are enthusiastically involved in myriad open source projects. They aren't generally thought of as having socialist tendencies.

HOW IS SOFTWARE BUILT?

http://www.multunus.com/2011/02/software-development-as-a-balance

Here’s a fairly typical example of the “software development lifecycle.” It lists a large number of activities and requirements, which deserve to be looked at in detail.

Whether all these activities are strictly necessary or not could be a pleasant discussion over drinks, but I get the definite impression that many open source projects aren’t engaged in any kind of formal process. This freedom, of course, is one of the things that attracts many people into the open source world, and I would be the last person to suggest that unnecessary work should be involved in software production. Nevertheless, open source projects are predominantly run vby programmers, and often all project members are programmers.

WHO BUILDS SOFTWARE?

http://www.ambysoft.com/essays/agileRoles.html

This is a typical conception of the way software is developed in the agile development world. There are many different ways to think about the development process. Arguably, none of them is specifically right or wrong. But when you think about commercial projects the size of Python, there will be many more people involved than just programmers. So Python, like most other open source projects, has to press developers into service to fill the other roles, or leave them unfilled.

WE need to think about whether this works to the ultimate advantage of our projects, or whether it is really a rather inefficient way to produce software.

SOME DEVELOPER ROLES

Above are just some of the roles that a large-ish software development requires. I would argue that lots of the time projects focus largely or even solely on the programming task, with some design work. Many of the other roles do not offer attractive work for developers, who therefore ignore the need to fill them, often to the project’s great detriment.

A prime example here is the Python documentation. Good as it is in comparison with many other open source projects (and it is good, because there has been a lot of hard work put in on it), it still sucks. Its organization is mostly based on a fifteen-year-old structure, back when Python was a smaller and somewhat different language. I still have no idea why, when I want documentation on the Python standard types (int, list, dict, and so on) I have to look in the Standard Library documentation. They are available without any need for an import statement, so why are they documented with the library?

Of course anyone who has used Python seriously for more than a year knows these things, and can usually find out where to look for the information they need, but this is entirely the wrong emphasis. The first need for documentation is when you are learning a language, and then the information you need should be readily available. Who is the advocate for a rewrite or reorganization of the documentation? Nobody in the current development team relishes such a task, because we don’t have technical authors and editors as a part of our community. And yet this issue causes learners way more pain than it should, and even turns people away from the language.

This brings me to my central thesis: if we don’t broaden our community to include such people we are unlikely to achieve the world domination we all jokingly feel that Python could achieve. I know from my work over the last eighteen years, and from visiting Python gatherings like this one all over the world, that Python is poised for massive growth. It would be possible, if a concerted effort was made, to make Python the first programming language of choice for literate students. The world has woken up to the fact that computational thinking is an important skill in today’s complex societies, and some feel that Python is its ideal expression.

So, do we want to slam the doors, say “No, we like things the way they are, we aren’t going to change”? I suspect that some developers will feel that way. But as a PSF director I have to be mindful of our duty to promote the development of the language and grow the international community, and so I have to force myself to look for deficiencies than can be rectified. This means looking outwards, and trying to include people who can fill the necessary roles.

The alternative, of course, is either to ignore the need for a role to be filled or to have someone fill it whose time would be better spent writing code. Neither is an entirely satisfactory solution.

BEING MORE INCLUSIVE

Sarah Novotny, one of the OSCON conference chairs, wrote of her experience at the 2011 conference: "We must challenge the idea that if someone really wants to use a piece of software, he or she will be willing to slog through half-written documentation, the actual code base and an unkind user interface."

This is an approach that I have tried to promote to Python developers, with rather limited success. I can think of several reasons why this should be so. Here are some classic signs that the status quo is being defended:

Considering the cost of switching before you consider the benefits
Highlighting the pain to a few instead of the benefits for the many
Focusing on short-term costs instead of long-term benefits, because the short-term is more vivid for you
Embracing sunk costs

We need to try and broaden our development community to include people with the necessary skills, even though we aren't necessarily very comfortable getting close to them. This makes sense to me as a historical perspective: back in "the old days", using an open source project was highly likely to involve you as a contributor as well. Nowadays there are many other roles for contributors to fill: bug reporters, technical writers, bloggers, testers, and I am sure all you core Python developers have your own ideas for roles that could reduce your workload.

David Eaves of Mozilla corporation has done a lot of work on how software communities can use metrics to improve their development process and monitor community involvement. He asks relevant questions like “why are bugs in this section of the code taking twice as long to be reviewed?” and “who has contributed consistently over the last 18 months, but not in the last 30 days?”

If we are prepared to answer questions like that we can take the information, both qualitative and quantitative, and then use it to continue to improve our communities.

INVOLVING THE COMMUNITY

We must accept that not every member of the community wants to be involved outside a fairly narrow specialism. This is fine with me: I would much rather have contented community members contributing their skills than seeing members pressed to leave their comfort zone.

At the same time, there is considerable evidence that an inability to contribute is a source of frustration to would-be active members. So if we want to grow our community we need to be welcoming and, dare I say, open. We need to make sure that the barriers to entry into our community are as low as possible, and that people whose ideas may not yet be well-informed do not find a hostile environment when they join.

The forthcoming revision of the python.org web site is an attempt to democratize the content and lower the barriers to making a contribution. The intention is to allow contributors to directly affect the content in their area of competence, without requiring mediation from a "webmaster" team who are familiar with the gory internal technical details of the site. More such initiatives are required.

REACHING OUT

So, it's important to reach out to the people with the skills to make our community, and our software, better: more user-friendly, more robust, better documented, and so on. The existing community does a fantastic job, but think how much better things could be if we added new skills by growing the community outside its current demographic. While one of the advantages of open source, to many participants, is its lack of corporate rules and regulations, we nevertheless should accept that we do face many of the same problems that businesses do. If we can find new and creative solutions to those problems then that's great. If we can't, however, we shouldn't necessarily ignore the solutions that have already been developed in more traditional environments.

Besides wanting to increase the size of the existing Python community, I would also like to see the Python world reaching out more determinedly to groups supporting specific applications areas such as accounting, law, the conference industry, the print industry, the transport industry, the aerospace industry, and so on.

The scientific community is a stellar example of the sort of collaborations I envisage. There, the community members are mostly not interested in Python for its abstract beauty, and don't have any ambition or desire to contribute to the core. They use Python because it has been demonstrated to be an effective tool for solving their problems. We need to remember that's the motivation of almost all users outside the development community.

In my secret life as a conference organizer I am using Python in the shape of Eldarion's Symposion project. While it does currently have many failings, my approach is to use it intensively, and feed back the problems to the developers for fixes. As funds become available we intend to invest in improvements that will make our job as conference organizers easier and more effective. I am hoping that eventually, once we have together built a project that non-specialist users can be happy with, Eldarion and The Open Bastion will find a market among the small to medium-sized conferences, and another industry will start to fall under Python's sway.

Repeating such efforts across many industries is, in my opinion, the best way to make Python popular, and its many existing successes are a stable platform on which to base this work. This will involve effort, since we have to explain open source to people who are not as familiar as we are with the milieu, and whose interest in it is limited to the areas in which it can help them get their jobs done.

TAKING RESPONSIBILITY

Python is a mature technology, and it's time to consider it, if not exactly a finished work, at least as something that can be presented as a feature-complete tool for use in many different application areas. The UK government, having accepted that its computer education has gone through a period in the "dark ages," has accepted that computational thinking is a core skill in today's world. It looks as though Python will be adopted aggressively as a teaching language. I hope that success will be mirrored in many other countries.

If we are prepared to engage others with different skill sets from our own, and listen to their needs, we can build new communities who will complement our existing skill set and help us promote Python world wide in a multitude of application areas. The benefits for the Python community, though, are only a tiny portion of the gains to be made.

I mentioned at the start of my talk that the world is demonstrably broken. Central governments have neither the resources nor the imagination nor the flair to apply global solutions to local problems. It seems self-evident to me that the best way to solve local problems is locally, and so we need to persuade these governments to release some of their control and their funding to the communities with the power to solve those problems.

Eric Sterling gave a very engaging keynote at DjangoCon this year in which he suggested that it was important for a much broader cross-section of the population to make their needs known. As a group who only have to see an itch to start thinking about how to scratch it, I think that open source devotees are in a position to provide some valuable answers to the question of how we fix some of today's complex problems. This means leaving their comfort zone, reaching out and engaging with government and commerce at all levels, and I want to persuade open source communities to use their considerable skills to do so. The world is broken, but I think with determination and goodwill we can help to fix it.

This is an exciting time to be a Python programmer!

I Should Have Known Installing Windows Would Be Painful

2012-10-12T02:35:00.002-04:00

As a geek I tend to have a bit of hardware lying around unused, so when a neighbor told me his laptop was starting to flake I suggested he might want to borrow one from me rather than put his wallet under further strain.

Two likely candidates emerged. One was a venerable Dell Precision M6300 and the other a Lenovo ThinkPad X6. I thought both had Linux installed, so I was unpleasantly surprised to see that the Dell didn't have a bootable image on it. Never mind, I thought, I can use the ThinkPad. Alas, the person who'd installed Linux hadn't given me a record of any passwords, so I called his father and I'm expecting a message back soon enough that it didn't seem worth rooting the machine (which in retrospect seems like the best option).

Bright ideas can hit at any time, and it was now that I remembered I had both the original install disks and a Windows 7 Upgrade set. My neighbor is actually a Windows user, so it seemed like a friendly act to install Windows for him. Since the disk had nothing on it there seemed to be little harm in requesting a complete installation, which I did. After answering a few questions it was grinding away, so I went out. When I arrived back a couple of hours later the installation was almost complete, waiting for the entry of my Windows product key. Which I duly entered from the Windows 7 Upgrade box. Product key invalid.

Being of a dyslexic nature, not to mention getting on enough to be experiencing some long-sightedness, these episodes are painful for me. Suffice it to say that after three further attempts involving the use of a flashlight and a magnifying glass I was positive the product key was entered correctly. So I figured since I was installing an upgrade maybe the Windows product key required was that of my original Windows installation. So all I had to do was turn the laptop upside down and peer again. The same performance with flashlight and magnifying glass was again repeated multiple times. All no go.

So there I was, with a stuck Windows installation. The only way out that I could see was to call Microsoft, who helpfully gave the support number in a little booklet in the Upgrade box. After I enter three or four digits to indicate I am having activation problems with Windows I am finally told that Microsoft are sorry, but they are now closed and I should call back during their normal working hours. They do not tell me what their normal working hours are.

I guess I'll call back tomorrow, and bite my tongue rather than pointing out you never have any of this nonsense with Linux. When I need support there, the best in the world are available on the Internet.

Links for 2012-09-18 [del.icio.us]

2012-09-19T00:00:00-07:00

Create Your Own @font-face Kits | Font Squirrel

Links for 2012-09-15 [del.icio.us]

2012-09-16T00:00:00-07:00

Links for 2012-09-10 [del.icio.us]

2012-09-11T00:00:00-07:00

Links for 2012-09-09 [del.icio.us]

2012-09-10T00:00:00-07:00

Links for 2012-09-08 [del.icio.us]

2012-09-09T00:00:00-07:00

Issue #44: Failed to validate oauth signature and token · abraham/twitteroauth · GitHub
$connection->http_header['date']
Single-user OAuth with Examples | Twitter Developers

Adafruit Pi Cobbler

2012-09-01T05:34:00.001-04:00

Here is a picture of the extension board I built in about ten minutes this evening.

The kit includes a ribbon cable to attach to the Pi, and the little circuit board plugs in to your prototyping board on which you can build the rest of your circuitry.

This will be useful for projects too large for the Proto Plus to accommodate.

Pi With Proto Plus

2012-09-01T04:54:00.001-04:00

Now I can blog straight from my phone I thought I'd post this picture of my Raspberry Pi, in its case with the lid removed and with my new Adafruit Proto Plus board in place.

The add-on board is well-designed, allowing you to easily tap the Pi's signals for use on the breadboard (an optional extra I couldn't resist) or for external circuits.

I have also bought a couple of Adafruit's Pi Cobbler kits, which will let me attach the fully-enclosed Pi to a larger breadboard. I am happy to say my soldering skills are returning despite a thirty-year hiatus.

Note: I am not impressed by the way editing a mobile blog post immediately requires me to insert paragraph separators in what until then appeared to be a well-formatted post.

Spam Report

2012-08-23T04:01:00.001-04:00

Spam still exists.

Noller Does It Again

2012-08-02T03:58:00.001-04:00

I mentioned in my recent OSCON write-up that I had the privilege of announcing that this year's winner of the Frank Willison Memorial Award was Jesse Noller. The most spectacular contribution to his win was the amazing PyCon that he chaired with record results: over 2,200 delegates and an amazing program opening with dancing robots that, coming only ten years after PyCon took its first faltering steps, hit me quite emotionally.

Jesse is once again PyCon chair, and he has just announced the most amazing addition to next year's PyCon. Delegates will be able to sign up their over-12 kids for a short course, using pygame, to help them understand why their moms and dads are so crazy about Python. Not only will the workshop be free, each registered kid will be given a free Raspberry Pi computer complete with an operating system on an SD card.

I'm so enthusiastic about this I have already approached the organizers, Lynn Root and Katie Cunningham, to offer my services as a classroom assistant. I encourage you to do the same if you enjoy helping kids to learn.

The conference is still eight months away, people. Heaven only knows what other fabulous treats he has in store for us all. Go, Jesse.

Question Question

2012-08-01T00:48:00.003-04:00

So, if the ladies of PyPy started their own organization would they be the PyPyLadiesLadies?

DjangoCon Comes to DC

2012-07-26T00:02:00.001-04:00

Last year The Open Bastion asked the Django community where they wanted to hold the next couple of DjangoCons (since this is a community event, we figured that it made sense to ask). The first answer from most people was “this place close to my home town,” and the second one from the Django Software Foundation was “you’ll have to decide.” So much for community consultation.

In fact we did interact with the denizens of the django-users list, and eventually settled on Washington, DC for this year and downtown Chicago for the next. That being the case, the next thing was to recruit the community volunteers. Fortunately this wasn't difficult, since for DC they essentially turned up in my inbox one day basically asking what they could do. So far they have arranged a great sprinting facility, and become involved in the financial assistance of impecunious delegates. It's great to work with an enthusiastic team of volunteers

People sometimes ask why a conference like DjangoCon uses volunteer help, and the answer to that question is complex. First of all, the volunteers can take care of areas we don't have the time or the local knowledge for. Those close to the conference are invaluable in performing on-site actions ahead of time. Secondly, their involvement retains the essential community feel of the conference. Our role is principally to ensure that the venue provides what it should and the logistics of the conference run smoothly (and thereby hangs a tale).

Anyway, with 39 days to go to DjangoCon we are entering the final checklist phase. Apologies to the sponsors, as we have been dilatory getting their details up on the site. We are working on it.

It's going to be interesting being in DC, to which I lived close for almost fifteen years. The hotel, thank heavens, is just around the corner from the airport—one of DC's advantages is that you can be downtown very quickly after clearing the airport. Just make sure you fly into Reagan National (DCA) and not Dulles (IAD) or Baltimore-Washington (BWI) unless you have to.

Now the schedule has been announced Early Bird tickets have been moving briskly, but it looks as though they won't run out before the Early Bird period ends on August 3. The new Supersaver tickets were popular, though not as many students bought them as we anticipated. There is something for everyone in the schedule, and we'll be featuring the keynotes in a separate blog post soon.

We have a code of conduct, not that different from the PyCon code of conduct. The principal value of such codes is in putting people on notice that certain standards of behavior are required.

One of the things I like most about the Python and Django communities is that the individual members of the community largely feel empowered to police such things informally. This means that most issues never flare up into the kind of ugly incident that we are all tired of reading about because such behavior gets nipped in the bud before it has chance to make serious trouble*.

Running these conferences is an effort, but whether or not they make money it's always gratifying to have people thank us for making the effort to run them. Now that I am no longer the chairman of the Python Software Foundation I am finding other ways to serve the community which has been a major part of my life for the last fifteen years. Making the transition for community leader to open source entrepreneur might not be easy (I have already made a couple of expensive mistakes). I hope I have your support.

* I realize there is an element of hubris in such assertions, but nevertheless wanted to underline that harassment is taken seriously.

Fun and Games at OSCON

2012-07-25T21:42:00.000-04:00

Once again I spent a week in Portland with over 3,000 Open source enthusiasts, developers, business people and community managers at the largest US open source convention. Once again the hallway track was supremely interesting, and there was much of interest in the sessions.

For me there was a little more business than usual, as a new client of The Open Bastion was present and this was an opportunity to learn more about their team, meet people, form impressions, find out how they work and so on. Also the Python language had a stand in the Open Source Pavilion (alongside many other famous open source names), and I took my Raspberry Pi along. It was amazing how many people had heard of the Pi, and very few actually had one. The commonest comment was “I’m still waiting for mine.”

Alas I again didn't have very much time for the excellent Community Leadership Summit which now traditionally precedes OSCON, though I did manage to take part in one session about conflict resolution, and heard much useful stuff in plenary talks. If you are the leader of any open source community group I would definitely recommend going to this (free) event. It's well worth the cost of an extra couple of night's accommodation. Monday was taken up with clerical work, but on Tuesday I did make it to several sessions of the Business Leadership Day tutorial, and found them sound to excellent.

Tuesday evening Camp OSCON was held—indoors, due to threatened thunder storms which in the end never transpired. This was a traditional O'Reilly party, open bars, many different types of food in all-you-can-eat quantities, airbrushed fake tattoos, the whole bit. I managed to score four party tickets for a friend who came with his wife and two children. They were excited beyond belief, and it was delightful to see how much pleasure a two-minute stop at the registration desk had created.

Among the many activities was a dunking stool, allowing three pitches in return for a $10 donation. My major part in the evening's activities was to be (as I thought) the final target of the event. I took my place, and was duly dunked several times. At one point a guy who looked a lot like Tim O'Reilly took a shot. Good heavens, it was Tim O'Reilly. After failing to unseat me with any of his three pitches he moved closer to the target. Saying “How does this work?” (something I suspect he was not in need of information about) he pressed the target and promptly subjected me to my fifteenth ducking of the evening.

Tim O'Reilly in the dunking tank
Photo from Flickr by aaronparecki

When I emerged from the tank I thought it was all over, but then I saw Tim getting into it. Even though I was still wet I couldn't resist, and promptly made the necessary donation to put me at the front of the line. I am happy to say by the flukiest luck I managed to turn him from dry to wet with my first pitch. Success! My two remaining pitches failed to hit the target, so I went off to the men's room to dry and change. When I emerged I couldn't resist going up to the target, saying “How does this work?” and subjecting him to a second dunking.

I went around back to pack my wet clothes and he insisted on reaching over to shake my hand with a big smile, which reassured me that there were no hard feelings and that the cancellation of my O'Reilly School of Technology classes was not imminent. Tim O'Reilly is a good sport.

Wednesday's keynotes were thought-provoking. I particularly like the fact that the open source world appears to be becoming less shy about the value that it creates in the economy. Tim O'Reilly suggested that successful organizations create more value than they capture, and was very scathing about the short-term thinking and transparent absence of business ethics on Wall Street.

He did some very interesting sums suggesting that open source is a huge net plus to the economy. He likened the situation to that in energy: if we use a clothes dryer to dry our clothes that energy is metered and measured and accounted for. If we dry our clothes on the line using solar power nothing ever gets recorded. Similarly the benefits of open source software have not been sufficiently trumpeted to the world, and so are taken for granted by the commercial enterprises that profit from them because they do not perceive any significant costs. He ended by thanking those of us who have been working in open source for a long time for our forward thinking and our contributions.

Much of the rest of the day I spent talking to people either on the Python stand or in the hallway track. It is always impressive to learn about the range of applications to which open source is being put. The medical faction are much in evidence. The most surprising discovery I made is that at the core of the award-winning open source VistA system is a MUMPS database, technology whose roots must go back over forty years. I sure hope there's a Python API.

Thursday morning's keynotes were more practical in nature, my favorite being the talk from Canonical's Mark Shuttleworth. I'm a sucker for technology, and his explanation of how the juju system lets you build and model complex, scalable systems on the desktop and then easily migrate them to the cloud for testing or production was quite something. The practical demonstration of migrating a service from Amazon to the H-P Cloud service was impressive* (though I was amused to see he did at one point get bitten by a minor gremlin).

There was a frankly disappointing interview with Microsoft's Senior Director for Open Source Communities, Gianugo Rabellino. Not only did it sound as though it might have been scripted by Microsoft's public relations department, it seemed to be mostly "sound and fury, signifying nothing." I am aware that Microsoft** are now a significant contributor to (for example) the Linux kernel and other open source projects, but it's difficult to have a company's representatives talking about open source in "snake oil" terms one year and then see them hold out the olive branch the next.

There is still a substantial mistrust of Microsoft's motivations and purposes among a part of the open source community, as evidenced by a subtle change in the audience's mood. They will have to work hard to prove that they deserve a place in the open source community. This is a pity, because Microsoft employ many smart technical people who fully "get" open source, but it is taking the corporate side a long time to come to terms with it.

The Python stand was very busy again, as were those of the other open source projects. The day went quickly, and I got to listen to a couple of track talks, but again a lot of my time was spent in the hallways. I also spent some time observing the O'Reilly and Convention Center staff, who appeared for the most part to be a model of curtesy and efficiency. In the evening I was invited to the chairmens' party thrown by co-chairs Edd Dumbill and Sarah Novotny. This was, as is often the case, a fairly low-key affair attended by many likable people, and I was surprised to see that it was after midnight when we left. Time does indeed fly in good company.

This meant I had to hustle and get my beauty sleep as I was a small part of Friday's keynotes, announcing a well-deserved Frank Willison Memorial Award for Jesse Noller. Jesse is the classic busy person you should ask to get things done. His work rate is phenomenal, and he held down a full-time job while putting together the most amazing PyCon ever and going through a good deal of personal disturbance. I can't think of anyone who deserves it more, and the PSF Board was in full agreement that Jesse should be the recipient.

It was fascinating to spend a little time backstage and see the technology and the production values that now support OSCON. The show is professional from its head to the tip of its toes. The keynote session was followed by more hallway track and meetings while the sessions continued, then back for the closing plenary session at which Open Source Awards were handed out and the final entertainment was provided by Paul Fenwick, always worth listening to even when his topic is light.

After the formal end of the conference I had a couple of business meetings and went home (using public transport, because I live in Portland, hooray!) for a well-earned nap. Feeling much better-rested the next day I was happy to host the inaugural OSCON Survivors' Brunch at the Tabor Hill Cafe, a local place I make sure to patronize while I am at home. This was kind of an ad hoc event, attended by about a dozen of my favorite open source people, and I will repeat it in a more organized way next year.

OSCON always leaves you with plenty of food for thought, and this year I was much more encouraged about the prospects of open source entering (or even, dare I suggest, becoming) the mainstream. Mark Shuttleworth revealed OEM connections this year and claimed that next year 5% of all computers will ship with Ubuntu pre-loaded. I'd recommend buying your ticket for next year's OSCON as soon as they come on sale.

* H-P were a major OSCON sponsor this year; this was a shrewd move
** Microsoft also sponsored OSCON

[Disclosure: Steve Holden's business runs technical events and conferences,
and Steve profits from the sale of O'Reilly School Python classes]

Back in the USSA

2012-07-14T00:34:00.000-04:00

My return to the USA after four weeks in Europe is to remarkably pleasant weather, which bodes fair for OSCON next week. It is something of a blow to be back in harness, but made more bearable by the thought of seeing so many friends again at OSCON, and so many more at DjangoCon—in Washington, DC for the first time this year. So at least I am working towards sympathetic ends.

During the trip I spent a week on vacation in a small town in Southern Spain. After an intensive week of packing and wrapping in London I then flew to Florence for EuroPython. I was charmed by EuroPython, and delighted (having been unable to attend last year) by Florence.

This was my first visit to Florence. Arriving from London's City airport wasn't terribly pleasant, as the city's fairly small airport is under renovation and the line for the passport check didn't even enter the terminal building for about thirty minutes. Fortunately I had not checked any baggage, so I was soon in a taxi once I left the terminal building. [Since I don't want to end on a sour note I will mention now that the outbound experience was also troubled by long albeit air conditioned lines, apparently due to computer systems issues; I wrote this down to teething troubles].

A €20 fixed cab fare took me to the Grand Hotel Mediterraneo, where the conference was already heating up—literally: the temperature was around 35°C (100°F) on that first day, and the weather was baking all week. The Italian organizers had worked hard to ensure a smooth flow to the conference, although the two-floor nature of the event inevitably meant that the staircase traffic was heavy at times. The air conditioning found it hard to keep up when rooms were full, and I sympathise with the venue—it is difficult to deal with such extreme temperatures.

My delayed arrival meant that I had missed Guido's “Not the State of Python” talk, at which I later heard he claimed he had used the same slides as a previous presentation but said completely different things. The BDFL was certainly relaxed and entertaining as he later answered questions from the community relayed by Harald Armin Massa. When accosted later in person he appeared to be in fine form, and although he is traveling a little less (the lucky man) he was due to go on to Zurich on business after the conference.

The program, primarily in English with one Italian track, was at all times worthwhile and often entertaining and stimulating. It was a pleasure to spend time with acquaintances old and new, and find out some of the amazing things that the Python community has been up to. The lightning talks were again chaired by Harald Armin Massa, now billing himself as “Lightning Talk Man” and I should have liked to see more than I did.

The Python Software Foundation was among the conference's sponsors, and ran a stand with the PSF's new conference kit on display for the first time. I spent some time on there with Marc-André Lemburg, Pat Campbell (the PSF's administrator) and Ewa Jodlowska (its Conference Coordinator), Quite a few people expressed an interest in the new associate member scheme, and we had new stickers to pass out to those who signed up. The ever-popular Python stickers also disappeared fast, and we handled inquiries about advertising in the new Python brochure whose production Marc-André has been leading.

This was a very special conference for me in many ways, as I ease back into community member status. I was able to spend some relatively relaxed time with old friends whom I rarely see. The food in Florentine restaurants is excellent, and each night was a different feast. At least one Italian admired my restraint in leaving a part of my secondi, the entrée dish. Little did he know that I was already at bursting point. Beware, portions tend to be generous, and the food so good that it is an effort to stop eating. After that, I approached conference meals with rather less gusto.

Henrique Bastos from Brazil

Over the weekend I was joined by a friend from the UK and with Henrique Bastos and his wife we took the open-topped bus tour of the city. A walk through the Boboli Gardens to the old town, across the Ponte Vecchio and past the Uffizi Palace saw us ensconced in a typical Florentine café/bar with a bunch of young guys who were happy to advise on food and drink. A taxi back to the hotel seemed advisable after a most diverting (though somewhat tiring) day.

On the Sunday, before returning to the UK we took the open-topped bus out of town and winding through the hills to Fiesole, a place to which I shall return to enjoy at greater leisure in future, I hope.

Fiesole, Florence, from above [courtesy of Google Maps]

Oh, and during it all people from (I'm guessing) over twenty nations got together and shared a huge amount of knowledge about Python, thereby helping to grow the Python community and bond it together through the forge of shared experience. What could be better than that?

Bravo to the organizing team for keeping everything running so smoothly. Those guys will be a hard act to follow, but fortunately nobody is going to have to try just yet. Yes, EuroPython will run in Florence for an unprecedented third year. My advice would be to book early.

Twelve Years a Blogger

2012-05-19T06:02:00.000-04:00

Some years were busier than others, but I am still here, and still finding the time to blog occasionally. I try not to be put off by (or unduly stimulated by) thoughts such as “It's been too long since I put something out” or “it's been so long now it would look lame to post something.”

If I have something I think is worth posting and (the kicker) the time to do it then I will by all means write something down. Just been a bit busy this lifetime is all ...

“Life’s what happens to you while you’re busy making other plans”—John Lennon

Open Django Central Now Open to Sponsors

2012-05-18T18:29:00.001-04:00

The Open Bastion, the subsidiary that runs events and conferences, has announced that its Open Django Central event, new to the calendar this year, is now (yes, we know) finally open to sponsors. Sponsorship is very reasonably priced, and the more money that comes in sponsorship the better we can make the event for the local Django community.

This represents a rare opportunity to connect with a diverse group of “djangstas” from the greater
Chicago and surrounding areas. The conference is targeted not just at technologists but also at
others wishing to engage with the open source world and find out what these technologies can do.

We hope that by adding your name to the roster of those supporting Open Django Central you will
indicate your enthusiastic support of and interest in open source technologies, in particular Django and Python. It's easy to ask about becoming a sponsor.

We are also still looking for speakers, so please let us know of anyone (you included, dear reader) who might be interested in giving a (technical, not marketing) talk at the conference. Slots are from 30-90 minutes.

And please act now, as time is (yes, we know) getting short!

Refactor All the Laws

2012-05-17T20:20:00.000-04:00

In the Python world we try to keep things simple. There are very good reasons for this. Brian Kernighan, a well-known programmer responsible for parts of the design of UNIX™*, once famously observed that

Debugging is twice as hard as writing code in the first place. Therefore, if you write the code as cleverly as possible you are, by definition, not smart enough to debug it.

This principle is not observed by programming beginners, who know enough to get themselves in trouble, but often not enough to get out of it. Anyone who has been coding a long time (and believe me, I have been coding a long time) knows not to get too smart lest they write code they cannot debug. It's slightly different when you are struggling to establish an architecture (are we a library or a framework?), but once the production environment is established you really want to be able to crank it out without having to rethink each corner case.

I even once went so far as to write a pre-processor for BASIC PLUS that operated on BASIC lines (which had to be numbered, whereas the input to the pre-processor did not) much as an assembler did to lines of symbolic machine code. In other words, I made BASIC code relocatable and made library sharing between projects much simpler. Because it was really more like a linking loader than an assembler (though it had features in common with both), I called it Blink. But that was thirty-odd years ago, when I was writing accounting systems. It certainly let us crank the code out. Happy days.

Of course Blink no longer exists. I am not one of those programmers who keeps every line of code they have ever written, regarding most of it as ephemeral: built to perform a task, and no longer relevant once the task is complete. I do not envy the curators of computer museums, who must decide what is worth keeping, and what can be kept running. You can do that with hardware, just about, but with software the profusion makes it impossible to track what's going on. Perhaps soon the open source world will find fixes for this bug—certainly the appearance of public DVCS systems will help. Some software, however, seemingly goes on for ever and ever.

It was very pleasing earlier this year to see a bunch of BBC Micros of ancient vintage still doing what they were designed for thirty years later. I wonder whether we'll ever see a 30-year-old iPhone anywhere but a museum? (To be fair, you don't see many BBC Micros nowadays). There are relatively few engineers building control systems and the like for a lifetime much longer than that.

Of course most systems of any age have been modified somewhat from their original purpose. You build a billing system, then the sales department come along and say "we could double our revenues if we could bill more flexibly," and the dance begins. You fix your programs to handle new requirements, they come along with even newer requirements, you add more fixes, and so on. Unless you are very, very disciplined, and are working with well-designed well-written code (e.g. if someone else wrote the program badly you may be screwed) you can end up finding that the change you make, while it meets the new requirements, no longer meets earlier requirements because of unintended consequences from your change. In simple terms, your program has become so complex that fixing it in one place breaks it in another.

In the software world we can use regression tests to alleviate the worst pain from this kind of activity (in addition to the unit tests we use to establish basic functions operate correctly). Whenever you find an error in the program, you write a test that fails with the problematic release but should pass when the system is fixed. This has the advantage that if your changes cause unanticipated failure then there should be a high probability that at least one existing test should fail. The presence of such regression tests sets a sort of “high water mark” for software performance. It has to be at least good enough to pass all the tests, or something is broken. In the presence of the tests we can refactor our code (reorganize and re-structure it)

Imagine now, if you can, a computer program two hundred years old**. Yes, I know, that pre-dates even Charles Babbage's analytical engine by more than a century. Never mind that. Just suppose that by some freak of probability some primitive computing technology had been developed by an unsung genius, and that its output is so valuable that it must be kept running†. The order of society itself depends on this program running, and yet it has never had a single test written for it. Don't blame the authors, when the constitution was written there was no such thing as test-driven development.

And yes, I am talking about the law of the land as something in need of refactoring. In just the same way as software engineering has benefited from test-driven development, so the law would, in my immodest contention, benefit from principle-driven development. By this I mean to suggest that the lawyers, when proposing a law, should list some desirable outcomes (tests) which somehow engender the law's purpose. If we could at least get agreement on what such principles might be, and the fact that they are desirable, then we might establish benchmarks for the operation of a law and be able to reject amendments that violate the principles (break the tests, in coding terms).

Just as sometimes we get our tests wrong, and have to rewrite them, sometimes legislators will occasionally get the principles wrong and need to rewrite those. But a discussion of principle would be a matter for all-out debate, whereas it seems to me that modifications of the law would be less contentious as long as none of the underlying principles were violated (in other words, as long as the law introduces no regression errors).

The present laws are full of special cases, introduced because of the lobbying of those with special interests or to suit one particular constituency. It's time we stopped placing so much emphasis on passing new laws and decided instead to add principles to the existing law so that we could start to detect more easily when the law started to diverge from society’s desires about the way it operates. In time the law could be cleaned up in much the same way as a crufty old program can be re-engineered to bring it in line with more modern requirements.

At present the law is a bug ugly ball of string, and there are many professionals making a good living finding and exploiting loopholes that operate to the advantage of their clients. We need more foresight, and we need a legal system that effectively says “this law cannot be amended, and is not intended to operate, to provide tax benefits to those who do not require them” or “this law cannot be used to the benefit of anyone with above-average income.” While this isn;t a perfect proposal, it would perhaps serve to focus people's interest on those who are specifically intended to benefit from the passage of particular laws, and the principles might over time become an accepted set of goals for new legislation.

When I think of how crufty code gets after just a few years I shudder to think what the law must look like from the inside. It's certainly obvious that the legislature has not been operating “by the pexxople, for the people and of the people.” It's time we changed that. Since I have no vote I'd appreciate it if my voter friends could execute this change at the first available opportunity.

* Merely one in a very long list of achievements, as any Internet search will reveal
** Or, perhaps, 236 years old
† Believe it or not, at the time of the "year 2k" panic some banks discovered they were running (in compatibility mode) some programs originally written in 1400-series autocode for which they no longer had the source. This would have been more surprising back in the days when banks were regarded as reliable and responsible. Happy days.

Swan Song

2012-05-06T03:51:00.001-04:00

Well, I don't really know where to begin. Finally, and somewhat unexpectedly, there is an end to the tunnel entitled “Chairman of the Python Software Foundation.” When I stood for election to the board this year my manifesto included a declaration that by the same time next year I would no longer be chairman. The board at my request asked for nominations for a Vice Chairman to prepare to take over from me at a planned point in the future.

As the inauguration of the new board coincides with the unfortunate necessity of consulting orthopedic surgeons I am availing myself of the non-profit equivalent of the "get out of jail free" card, and handing over to the new chairman Van Lindberg (currently Vice Chairman, and the careful steward of PyCon in 2010 and 2011), somewhat earlier than originally anticipated. This will allow me time to rest and discover what comes next. Fortunately I don't anticipate any real problems, but it is good to have a Chairman waiting in the wings, supported by a Board that will continue to take capable care of business while I pay attention to other things. I know that the whole Python community will be behind them.

To those who wish to email me about PSF matters, please continue to do so if you don't know who else to contact (hint: our administrator and secretary Pat Campbell stands ready to help at all times). I will continue to try and ensure that all mail reaches an appropriate (i.e. not /dev/null) destination. Please do all you can to help the new chairman and board to help you. I am sure that the Foundation will under their leadership continue to move from strength to strength, reinforcing its mission of promoting the Python language and encouraging the growth of the international Python community. I remain on the board for the remainder of my elected term, and look forward to helping the new team strengthen the ties we have forged with the widest possible Python community.

Thank you for the opportunity to serve you. Good luck, Van, and over to you.

Links for 2012-05-05 [del.icio.us]

2012-05-06T00:00:00-07:00

Delicious.com - Discover Yourself!
You know, it would be really nice if bit.ly and delicious were better friends.

To My Followers

2012-03-06T22:26:00.001-05:00

Wow. Didn't even know I had any. Sorry about that. Been busy. Should get a few posts out during PyCon.

Help Wanted

2012-01-03T08:44:00.001-05:00

If you are here because you think this post might be advertising a job, please accept my apologies. If, on the other hand, you are looking for work, this might be the very place for you. If you don’t see the difference between “a job” and “work” then there might be something to learn here.

“Hold hard,” I hear you say (I have a very vivid and somewhat eighteenth-century imagination). “Surely that is a contradiction in terms?” (My New Year’s resolution to be less pedantic stops me from pointing out that you have ended a non-interrogative sentence with a question mark)(and my pedantry neatly negates the resolution). I take instead the consultant's approach of arguing that it depends what you mean by ‘job’ and ‘work’.

If you take your definitions from the market economy then a job and work are often pretty strictly equated. This point of view would not cut much ice with many of the people I know, though. I am delightedly sharing the company of many of them at PyCon this week* who work in open source, though. In their case the work they do on behalf of the Python community, and other open source communities too in many cases, is often done outside of their working lives. Admittedly many of the have the smarts to make a halfway decent living by exercising the same or related skills professionally, but that isn't necessarily a bad thing..

If you don't really understand how the open source ‘ecosystem’ works you might be fascinated to learn that huge swathes of the technology that is used every nanosecond to process unthinkable volumes of information across the Internet was created by people with the necessary skills who also happened to believe that it was more important to create the new technologies than it was to organize profitable industries around proprietary solutions that were the sole responsibility of single organizations. Those people, the kind of people who put their effort where their mouth is, are people I will support. And have, most recently as chairman of the Python Software Foundation.

Open source devotees generally seem to me to try and avoid organizational issues or involvement. This is perhaps understandable: why would people who value logic and reason highly want to involve themselves in environments where decisions are frequently made on irrational grounds? “To make the decisions more rational” is one answer. But if that was the only advantage of involvement it would asymmetrically benefit the irrational, and why should the rational waste their time? (Idiocracy, anyone?) Most open source programmers would rather code than consider strategy and tactics for a non-profit, even if the non-profit is supporting their principal programming language. That's human nature, and we should be grateful for it: coders gotta code, and the rest of us benefit from that. But it can leave a gap if we don't want open source to simply become the plaything of market forces.

The point is, some of the things business does, and some of the ways business organizes, are sensible for any enterprise of moderate size. Allegedly, corporations are legally people (and the band played “Believe it if you like”). Most businesses currently pay far too little attention to their social responsibilities. If people went around behaving like corporations do then they would soon get what in London used to be referred to as “a swift punch up the bracket”. One of the strengths of a market economy is supposed to be that the market can be changed by mass behaviors. There doesn't seem to be much sign of that in the banking industry at present, despite public confidence in them being at an all-time low. And who can be surprised about the lack of confidence from a populace whose equity has just been looted by rapacious bastards who see bonuses in the billions as their right?

At the moment there is an uneasy tension between commercial organizations and open source, although that has not stopped both sides from profiting by its adoption. I would like to think that while the non-profit side can learn from commerce it can also teach commerce a few things about how to manage technical resources. We can, at least, try leading the horse to water. If it won't drink then I say let the damned thing die of dehydration. The signs are very clear that the present order can only be maintained by increasing levels of brutality (“the floggings will continue until morale improves”). Adapting to change was never the forté of the dinosaurs.

On the principle of “if you can't beat them, join them,” I think it's time for the open source movement to admit a few things. Such as: not everything can be done with software; foundations need skills that aren't readily available from the geek population (whether through lack of ability or inclination is hardly material). The Foundation I chair is likely, from existing evidence, to be receiving a fair amount of support as the economy recovers—and no, I do not claim to know when that will be. This will mean that more things need to be done, which in turn means more administration and accounting—two things I am as bad at as many of my acquaintance are. The inevitable impact of scaling up the organization is to require more of the kind of services that geeks (for want of a better or more descriptive term) are ill suited to providing. So either things will get clogged up or we will have to somehow involve more people who can provide the necessary “non-geek” skills.

Some time ago the Foundation hired an Administrator, who is now employed full-time, in acknowledgment of the need for such a skills base. I am happy to say that our Adminstrator and Secretary Pat Campbell has become an indispensable part of the Foundation. She continues to prove her merit by making things happen with no geek input! We cannot necessarily add staff at will, however, because we don't want to be one of those non-profits that exists to pay exorbitant salaries to its management for very little reason. Contributions should primarily go towards funding the mission.

There are many other areas in which we are stretched, and perhaps surprisingly many of those roles do not require much in the way of geek skills. Our Treasurer, dealing valiantly with increasing volumes of financial data as our operations grow, could certainly use assistance. We need help with marketing, to encourage people to join and pay subscriptions for the privilege of being associate members of the Foundation. When the updated web site technology is deployed (still some way off, but definitely more than a gleam in the chairman's eye thanks to the efforts of Jesse Noller) we will also need people who can manage the large and growing volume of content.

All this is exceptionally good news for anyone in the "open source industry," and people with Python skills in particular. Python is so hot now it's difficult to believe. Major companies have stood inline to join up as PyCon sponsors, and the money this has brought in helps us to keep the cost down. People are looking for Python training at many different levels, and need help establishing programming best practices and the like if training is not your métier.

The Python community as a whole is such an incredible resource that it's difficult to imagine a skill we might need that someone doesn't possess in at least some measure. But maybe it's time to admit that in reality in job X we could probably only deliver 79% of the performance we would like to, and that has to address specific functions. If we can find a volunteer for Job X we release a developer to program; if the volunteer only happens to do an 80% job that's still a win.

* No, none at all, sorry. PyCon has been turning people away for weeks now.

This post has been in draft for some time, hence the screwy dateline. Don't ask me why Blogger does this.

Nothing Stranger Than This?

2011-12-16T01:02:00.001-05:00

I’ve just been reading Stack Overflow’s Strangest Language Feature thread, where I discover that (at the time of writing) the highest-voted Python complaint is actually about Java, but the complainant claims that JavaScript and Python are the same. So after I've got this off my chest I'll perhaps go back and look for the top complaint specifically about Python. In Java the complained-of code looks like this:



    try {

        return true;

    } finally {

        return false;

    }

This naturally translates into



    try:

       return True

    finally:

       return False

which, under every recent CPython I've tried (2.{6,7}.X and 3.X), returns False. What I can’t understand is why anyone would expect anything different. If the finally suite contains a return then it’s pretty apparent that the programmer of that function wanted the function to always return a specific value. If there had been code following the return statement would the complainers have expected that to be executed? I'm not sure I see the reason for the complaint at all. What would the complainers have this function return? What is the return statement supposed to do?

This gave rise to a further thought: does a return in a finally clause suppress any active exception? This is only one of a number of exception-handling issues I've been thinking about lately, kind of noodling around the corner cases. More later.

Competition: Working to Keep You Enslaved?

2011-12-04T15:37:00.000-05:00

The Digital Millenium Copyright Act was never my favorite legislation. It was the first of a number of moves that threatens the freedom of the end users of intellectual property (by which under many circumstances I refer to those who have paid good money for their real or virtual goods or services) to freely enjoy that which they have purchased. Since the Python world generally prefers not to place onerous licenses on users this isn't generally an issue in my working life, but it affects other aspects of it. Those other aspects are shared with the other millions on the planet who use a smartphone.

For all the brouhaha there's been about occupations this year (and I for one regard them as a welcome sign that Americans have realized that the future of democracy is up to them) the most insidious occupation I can think of is the one organized by wireless telephone carriers against their customers. With the aid of an application called “Carrier IQ” many handsets, supplied with software packages approved by their respective carriers, have been equipped with the provision to spy on the holder.

This issue was brought to mind by the news that the Software Freedom Law Center have filed paperwork with the Library of Congress seeking an exemption to the DMCA allowing smartphone users the full legal right to run any damned software they choose on their handsets, with or without the approval of the carrier (who is naturally in most cases the supplier of the handset).

I’ve not yet heard of any case where Carrier IQ has been deployed to the detriment of the phone’s owner, but I haven't done extensive research in that area. As someone who teaches security I know that a capability, once created, is highly likely to eventually be used, and often for unintended purposes—if only by those whose approach to moral and ethical questions is one of expediency. Thank heavens so few such people exist.*

This raised for me the broader question of why, in a society that lauds the value of competition, are wireless carriers allowed to sell ’phones that are locked to their networks? If the argument is that doing so allows them to use the profits from the handset to subsidize the contract, my answer is thanks very much, I will pay for a handset that I can take to another supplier if they offer better services, and you can reduce the price of my contract since you do not have to subsidize my handset purchase, thank you very much. And I want the best possible price on the ’phone, please, or I'll buy it from Mr. SmallerProfits round the corner.

By all means let the corporate giants carry on doing business as they are, but also please require them to supply unlocked handsets on request. That way I don't have to suffer weeks in Europe where I am forced to use an international roaming plan that charges me a minimum $100 a month on top of my regular service charge for a parsimonious 70 MB of data and something ridiculous for each few measly bits after that. Haven't they heard we live in the information age? Colleagues from other countries simply re-SIM their telephones locally and pay $30 for a card that gives them 2GB of data and 2 calling hours, topped up electronically by credit card.

It causes me to smile sometimes when I hear Americans blindly praising their country for properties it does not possess, or for being best in the world at activities in which they are in fact well down the league table (mathematics and science education being among those activities). To my mind, America is quite admirable enough in reality without the support of false information. I chose to live here, after all. But when I discover that Israel of all countries (and no, we won’t get into that here, thank you) has made it illegal to sell locked handsets, I have to wonder how much that particular legislative concession would cost the cellular companies in the USA and why the move hasn't been emulated here. I am sure the lobbying costs expended against such a change would be formidable. Times are good in the boardrooms right now. As the UK magazine Private Eye says, “treble brandies all round”.

So although I was delighted to see Verizon** issue a categorical statement that none of their devices have ever had Carrier IQ installed, their roaming plan sucks at least as badly as any I have come across. The fact that the charges are so much higher than those of the native carriers smells of anti-competitive practice to me. If competition is so good, why don't they all use handsets that aren't crippled?

So rise up, sisters and brothers, and occupy your handsets. You have nothing to lose but your roaming plans.

* For American readers, this is an irony alert. Or am I being sardonic? The real point is that some people just cannot be trusted. You know who you are.
** The current carrier for both my voice and data cellular services

Links for 2010-12-26 [del.icio.us]

2010-12-27T00:00:00-08:00

gMap - Google Maps Plugin For jQuery | Examples
Great code for interfacing with Google Maps wiyh jQuery