Basic aggregation

Included in MongoDB are three operators useful for doing basic multi-document queries and aggregations: count(), distinct(), and group().

Count

count() is most basic. count() returns the number of documents that match the cursor's query. To illustrate this, first we'll create some documents in a test collection:

>>> import random
>>> import pymongo
>>> conn = pymongo.Connection()
>>> db = conn.agg
>>> db.data.insert([
...     dict(x=random.random(), y=random.randint(0, 10)) 
...     for i in range(50)])
[ObjectId(...)]

Now, let's count the documents where y is 3:

>>> >>> db.data.find({'y':3}).count()
3

This is normally what we want, but what if our cursor is using a skip() or limit()? count() ignores those:

>>> q = db.data.find({'y':3}).skip(1)
>>> q.count()
3
>>> len(list(q))
2

If you want to take skips and limits into account, you'll have to do some (a trivial amount) of calculation yourself:

def the_real_count(count, skip=0, limit=None):
    result = max(0, count - skip)
    if limit is not None:
        result = min(result, limit)
    return result

Now we can use it:

>>> the_real_count(q.count(), 1)
2

Distinct

Another thing that's often useful is checking to see which distinct values a query returns. For this, MongoDB provides the distinct() method:

>>> db.data.find({'y': {'$gt':3}}).distinct('y')
[10, 6, 5, 7, 8, 4, 9]

Note that you need to provide the field you want the distinct values for.

Group

With group(), we can perform somewhat more complex calculations. Grouping in MongoDB is basically equivalent to a reduce() operation in Python; an initial value is provided, and then all documents in the group are reduced into it using the reduce function. The actual group() function takes a number of parameters, most of which must be specified:

key

This can be one of the following:

• None, indicating that the entire document is the group key
• A list of property names indicating the group key
• A Javascript function which will return the group key when called for each document

condition

This is a condition used to filter the input to the group operation (i.e. it is equivalent to the WHERE clause in a SQL statement GROUP BY, not the HAVING caluse.

initial

This is the initial object which will be used to 'kick off' the reduction.

reduce

This is a Javascript function which will be used to combine all the matching documents pairwise until only one remains.

finalize

This is an optional Javascript function which will be called on the result of the reduction to generate the final value. It might useful for computing a mean value from a sum and count, for instance.

An example is illustrative. To see the count and sum of all the x values in our collection, grouped by y values, only for y values greater than 3, the query would be the following:

>>> key = [ 'y' ]
>>> condition = { 'y': { '$gt': 3 } }
>>> initial = { 'count': 0, 'sum': 0 }
>>> reduce = 'function(doc, out) { out.count++; out.sum += doc.x; }'
>>> db.data.group(key, condition, initial, reduce)
[{u'y': 10.0, u'count': 6.0, u'sum': 3.9350696527378295}, 
 {u'y': 6.0, u'count': 10.0, u'sum': 5.312402119253351}, 
 {u'y': 5.0, u'count': 3.0, u'sum': 0.9414083462732217}, 
 {u'y': 7.0, u'count': 7.0, u'sum': 2.992867324959482}, 
 {u'y': 8.0, u'count': 6.0, u'sum': 1.429589852438617}, 
 {u'y': 4.0, u'count': 5.0, u'sum': 1.7804494395579495}, 
 {u'y': 9.0, u'count': 3.0, u'sum': 1.6105657361868966}]

If we would also like to return the mean x value, we can include a finalize function:

>>> finalize = 'function(out) { out.mean = out.sum / out.count; }'
>>> db.data.group(key, condition, initial, reduce, finalize)
[{u'y': 10.0, u'count': 6.0, u'sum': 3.9350696527378295, u'mean': 0.6558449421229716}, 
 {u'y': 6.0, u'count': 10.0, u'sum': 5.312402119253351, u'mean': 0.5312402119253351}, 
 {u'y': 5.0, u'count': 3.0, u'sum': 0.9414083462732217, u'mean': 0.3138027820910739}, 
 {u'y': 7.0, u'count': 7.0, u'sum': 2.992867324959482, u'mean': 0.42755247499421173}, 
 {u'y': 8.0, u'count': 6.0, u'sum': 1.429589852438617, u'mean': 0.23826497540643618}, 
 {u'y': 4.0, u'count': 5.0, u'sum': 1.7804494395579495, u'mean': 0.3560898879115899}, 
 {u'y': 9.0, u'count': 3.0, u'sum': 1.6105657361868966, u'mean': 0.5368552453956322}]

Warning

You might be thinking that group() is awesome, and it's exactly what you need, but there are two things to be aware of with group() which may make it a non-starter for your needs:

• group() cannot be used on sharded collections. It just doesn't work. So if you're doing sharding, you'll have to go the mapreduce() route (below).
• group(), since it uses Javascript, uses the SpiderMonkey global interpreter lock, meaning that all Javascript, whether it's a db.eval call, group, mapreduce, or $where in a query, must be serialized. So don't plan on doing a lot of group()s concurrently.

If you're only using group() occasionally on a non-sharded collection, it might be fine. For a more general and (IMO) better solution, however, read on for mapreduce:

Map/Reduce

MongoDB provides an implementation of the MapReduce algorithm using the collection-level mapreduce method. Once again, we're using Javascript, so the same caveats apply regarding the global Javascript interpreter lock, but since mapreduce provides support for sharding, we can actually squeeze a bit of parallelism out of it here.

One way to think of mapreduce as used in MongoDB is that it is a generalized group method. The particular differences between mapreudce and group are as follows:

• mapreduce allows us to provide an initial map function which will be used to generate documents to be grouped by key, whereas group only allowed us to specify a method that returned the key for a given document.
• mapreduce allows us to keep our results in a collection in the database, rather than returning them immediately the way group does. With mapreduce, we also have great flexibility on how those documents are stored in the collection. We can do one of the following:
• store the results in their own collection, obliterating any existing collection of the same name,
• store the results in a possibly pre-existing collection, overwriting any results that are already there, or
• store the results in a possibly pre-existing collection, reducing the results with any results currently stored there.

So to take up our example from the group discussion above, we can do the same thing in mapreduce as follows:

>>> query = { 'y': { '$gt': 3 } }
>>> map = 'function() { emit(this.y, { count: 1, sum: this.x}); }'
>>> reduce = '''function(key, values) {
...     var result = { count:0, sum: 0 };
...     values.forEach(function(value) {
...         result.count += value.count;
...         result.sum += value.sum; })
...     return result; }'''
>>> finalize = '''function(k, v) { 
...     return { sum: v.sum, 
...              count: v.count,
...              mean: v.sum/v.count } }
... '''
>>> db.data.map_reduce(map, reduce, {'inline':1}, query=query, finalize=finalize)
{u'counts': {u'input': 40, 
             u'reduce': 7, 
             u'emit': 40, 
             u'output': 7}, 
 u'timeMillis': 4, 
 u'ok': 1.0, 
 u'results': [
   {u'_id': 4.0, u'value': {
     u'count': 5.0, 
     u'sum': 1.7804494395579495, 
     u'mean': 0.3560898879115899}}, ...]}

Note in particular that the structure of our documents has changed. With group, we were working on 'native' documents from our collection. Now we are working with "keys" and "values", where the "keys" are whatever was the first argument to emit in our map fuction, and the "values" are whatever was the second argument.

The actual mapreduce method takes quite a few parameters in addition to the required map, reduce, and out:

map

required The Javascript function that calls emit(key, value) for each input document to the reduce. Note that in the map function, this refers to the current document being mapped.

reduce

required The Javascript function that takes a key and an array of values and which should return the reduction of those values. Note that reduce may be called multiple times for a given key, and may be used to further reduce an already reduced value, so the value returned from reduce should be similar in structure to the values emitted from map.

out

required (at least in Python) An object specifying what to do with the results, whether to return them inline, or to put them into a collection, and how to handle putting them into a collection. Options are listed below.

query

optional A MongoDB query used to filter the collection before sending its documents to the map function.

sort

optional This will cause the input to map to be sorted. Specifying a sort key that groups the documents by the key that map will emit() can lead to fewer reduce calls. Strictly a performance optimization unless limit is used, below.

limit

optional, not sharding-compatible Limits the number of documents send to map.

finalize

optional The Javascript function called on the results of reduce. Useful, for generating secondary aggregates such as mean.

scope

optional Used to set the scope variable for map, reduce, and finalize

sharded

If True, then shard the output collection on its _id field.

There are a few others, but they're less commonly used. The official docs give more details.

The out parameter is one that deserves a bit more explanation. It specifies what to do with the result of the mapreduce job, and can be of one of the following formats:

{'replace': <collection name>}

The output is inserted into a collection, atomically replacing any existing collection with the same name.

{'merge': <collection name>}

The output is inserted into a collection, overwriting any values with the same key but not removing the entire collection as replace does.

{'reduce': <collection name>}

The contents of the collection are treated as inputs to the reduce step. This option allows for incremental updates to an output collection.

You can also specify that the result should be written to another database using the the out parameter. For this, the order of keys matters and if you're using Python, you should use a collections.ordereddict or bson.SON object:

>>> db.data.map_reduce(map, reduce, bson.SON([
...     ('replace', 'agg_result'), 
...     ('db', 'otheragg')
... ]), query=query, finalize=finalize)
Collection(Database(Connection('localhost', 27017), u'otheragg'), u'agg_result')
>>> conn.otheragg.agg_result.count()
7
>>> conn.otheragg.agg_result.find_one()
{u'_id': 4.0, 
 u'value': {
     u'count': 5.0, 
     u'sum': 1.7804494395579495, 
     u'mean': 0.3560898879115899}}

Mapreduce and Sharding

If you need to get any parallelism out of your mapreduce, you're going to need to do it on a sharded cluster. If your input collection is sharded, then MongoDB will send a mapreduce job to each shard to be executed in parallel. Once all the shards are done, a final reduce/finalize will be run and the output collection will be written.

The details on how this happens in different versions vary, but the short answer is that if you're using MongoDB versions under 2.1/2.2, you should not shard your output. In 2.2 and above, sharded output of mapreduce has been overhauled and has much better performance/parallelism.

So what do you think? Anyone using any of the aggregation features in MongoDB? Interested in doing so? I'd love to hear about it in the comments below.

Subscribe to our newsletter

Changing things up a bit, today I have a special treat, as Jay Kelkar, an avid Pythonista and fellow member of the Python Atlanta user group, has agreed to write a guest post reviewing the web2py Application Development Cookbook. If you give him half a chance, Jay will be happy to expound upon the benefits of Web2Py, so this review seemed a perfect fit for him. So without further introduction, here's Jay's review of web2py Application Development Cookbook.

For years I have been focused in making things easier for users while putting attention on important matters . I have been interested in all things Python for over 10 years. There are many Python frameworks available and most are libraries with suggestions on how to use them to create web applications. Then web2py came to my notice, a Python web framework which does things quite differently from other popular web frameworks. The two features that put it above the others are: the developer deals more with the problem at hand and rather than the framework, and the built in ability to allow switching between major databases with very little effort. I have been using web2py for over 3 years and have written several major applications with web2py. I wondered if I would find in the book anything that would make me sit up and take notice. So it is with this in mind that I review web2py Application Development Cookbook by Mariano Reingart, Bruno Cezar Rocha, Jonathan Lundell, Pablo Martín Mulone, Michele Comitini, Richard Gordon, and Massimo Di Pierro (the lead developer for web2py).

Web2py is unique in many respects and what is positive to one programmer may appear negative to another. Some of the distinguishing characteristics follow. It is based on the MVC principle - which states that the Model which describes the data and its access, the View which displays or renders the data, and the Controller which manipulates the data, are kept separate and distinct. Unless you want to do something out of the ordinary, no modules need to be imported. It's ORM is distinctly SQLish, so anyone familiar with SQL will feel completely at home. Another is that it will automatically update the database schema to the model on request - this can be desirable in development. It can host multiple apps under one server. In addition, it requires no installation and is ready to run after it is unzipped into the target directory.

Subscribe to our newsletter

• Just use the procedural layer (Huzzah! Low coupling is good)
• Use the OOP layer through composition (OK, better than reinventing the wheel and coupling is still fairly low when using composition)
• Use the OOP layer through inheritance (Eek, coupling is increasing substantially now, but it's typically still better than copy-paste-edit style reuse)
• Use the upstream implementation as a reference or starting point when writing your own (coupling drops back to zero, but the line count of code that is directly part of the current project just went up substantially)

Apparently, in Python, it is easier to ask for forgiveness rather than seek permission.   That is to say, the normal approach when writing Python code is to assume that what you are trying to do will work properly.  If something exceptional happens and the code doesn’t work the way you were hoping, then the Python interpreter will tell you of the error so that you can handle that exceptional circumstance.  This general approach, of trying to do something, then cleaning up if something goes wrong is acronymically called EAFP (“easier to ask for forgiveness than permission.  Here is a (somewhat silly) example:

>>> a= 1
>>> b="2"
>>> a+b
Traceback (most recent call last):
File "", line 1, in
TypeError: unsupported operand type(s) for +: 'int' and 'str'

What has happened is that we have tried to add a word (ie the string “2″) to a number (the integer 1).  Addition doesn’t make any sense in this circumstance, so Python “throws” an “exception”.  In this case the exception is called a TypeError.   The problem with this is that when an exception is raised, then unless the program deals with the exception, the Python interpreter will step in, stop the program and print an error message like the one above.

Python deals with this by the try/except structure. You try to do the statements in the first block of code, and if there is an exception (that is, you failed when you tried), then you do the statements in the second block of code.

>>> try:
...    print a+b      # this is the first block of code and can be multiple statements
... except TypeError:
...    print "Can't add those things together!" # this is the second block of code
...
Can't add those things together!

Can you see that no exception was raised here?  As far as the Python interpreter is concerned, everything ran smoothly.  The program tried to print a+b, and, in doing so, tried to work out what a +b is.  However, it failed because a is a number and b is a string.  Because it failed nothing got printed in the first block of code.  Also because the specific failure was a TypeError, the second block of code was run.

A short code snippet can show you how this works:

>>> c = [1,"2",2,3,4,5]
>>> sumOfc = 0
>>> # Try Number 1 - this will fail
>>> for number in c:
...     sumOfc += number
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
TypeError: unsupported operand type(s) for +=: 'int' and 'str'
>>> # Try Number 2 - this will work
>>> sumOfc = 0  # need to reset the sum
>>> for number in c:
...     try:
...        sumOfc += number
...     except TypeError:
...        pass
...
>>> sumOfc
15

In the first try, the interpreter ran into the string “2″ and didn’t know what to do, so gave up.  In the second try, the interpreter knew exactly what to do if there was a problem with the addition (pass – or, in other words “do nothing”[see note 1]) so it went through the whole of the array adding up the things that were numbers without complaining.

Of course, you don’t use try/except blocks around everything.  You will usually have an idea about the parts of the code where something goes wrong.  You can put that code in a try/except block, with code to deal with a failure.  What you shouldn’t ever do is this:

>>> sumOfc = 0
>>> c = [1,"2",2,3,4,5]
>>> for number in c:
...     try:
...        sumOfc += number
...     except:   # <- Don't do this!
...        pass
...
>>> sumOfc
15

It is not entirely obvious what is wrong here – the code works just as well as before.  The only difference is in the except line – there is no exception specified.   The reason that this is bad isn’t that the code doesn’t work well, but, rather, that it works too well.  Every exception will be caught here, not just the one that you were expecting.  This is bad because if some other problem arises in your try block, you’ll never learn about it and your exception code will probably deal with it incorrectly.  You won’t know why your program doesn’t work because the interpreter won’t tell you.

The other approach to dealing with possible errors is called Look Before You Leap (LBYL).  LBYL involves making sure everything is right before you actually do the thing you do the problematic operation.  So, for example, you might check that what you were trying to add was an integer:

>>> c = [1,"2",2,3,4,5]
>>> sumOfc = 0
>>> for number in c:
...    if isinstance(number,int):
...       sumOfc += number
...
>>> sumOfc
15

Here the isinstance() function is another feature of Python introspection.  It tests whether the variable number is an integer (“int”).  So this code doesn’t even bother trying to add the number unless its an integer.  It first checks “is this an integer I see before me?”  If so, it adds it, if not, it ignores it.   Which is fine as far as it goes…

The problem with this is that it’s the wrong way around.  You’re not really interested in whether or not the thing you’re adding is an integer.  Rather, you’re interested in knowing whether addition is meaningful [see note 2].  In fact, this way of approaching things is broken:

>>> c = [1,"2",2,3,4,5.2]
>>> sumOfc = 0
>>> for number in c:
...    if isinstance(number,int):
...       sumOfc += number
...
>>> sumOfc
10

We’ve changed the last number in the array to be 5.2, but this causes it to be ignored because it isn’t an integer – ooops!  Applying the earlier try/except code gives the right result though:

>>> c = [1,"2",2,3,4,5.2]
>>> sumOfc = 0
>>> for number in c:
...     try:
...         sumOfc += number
...     except TypeError:
...         pass
...
>>> sumOfc
15.199999999999999

Well, accurate up to the rounding error…  Our try/except code gave us floating point addition for “free”, where the LBYL failed.  In fact, it’s even more interesting because how it works is dependent on what we define sumOfc to be:

>>> sumOfc = ''
>>> for number in c:
...     try:
...         sumOfc += number
...     except TypeError:
...         pass
...
>>> sumOfc
'2'

Wow! If that didn’t make you giddy with excitement, I don’t know what will.  The same code works if we make sumOfc a string.  Implicitly then we’re asking Python  to join the strings (which is what + means for a string) together and ignore stuff that can’t be joined.   We got this entirely for free from our try/except structure, something that would have needed reworking in the LBYL.  In fact, we could use this structure for any object that has addition defined for it.  It’s this sort of clever which makes Python so good.

Caveat:

Normally the place to use exceptions is where, every once in a while, something out of the ordinary will happen and you have some idea about how it will be out of the ordinary.  In the first case, if it’s not unusual then it’s not exceptional – don’t handle it with exceptions.  In the second case, if you can’t identify how it will be out of the ordinary you can’t deal with it in your except block.

Notes:

1. The point of the except block is to deal with any problems in the try block.  In this case you might try to convert the value to an integer in order that it could be added.

2. Implicitly we’re talking here about breaking interfaces. You, as a programmer, shouldn’t be concerned with what is going on under the hood.  You should be able to rely on the fact that an addition method is defined for an object.  If it is, then  you ought to be able to use it without having to know the internal details of the object.  So in the example here, addition is meaningful between integers and floating point numbers, but addition is also meaningful between two strings.

This is a bug that has been fixed in Python 2.7+, but it wasn’t backported for Python 2.6.

So if you build your own Python 2.6 binaries on Ubuntu 12.04 you’ll see errors like this when attempting to use anything SSL related:

*** WARNING: renaming "_ssl" since importing it failed: build/lib.linux-x86_64-2.6/_ssl.so: undefined symbol: SSLv2_method

Some of us still need Python 2.6, so I forked Python 2.6.8 and removed SSLv2 support. Tests pass and SSL works.

You can also just grab the diff below:

Update: M2Crypto requires patching as well:

We successfully recorded this installment! If you just want to watch it, scroll all the way down. Otherwise, read on for a short description of the projects shown off this time.

MDN Dev Derby: WebSockets

groovecoder plugs this month’s MDN Dev Derby. Dev Derby is a monthly web developer challenge on the Mozilla Developer Network. This time, it focuses on WebSockets and groovecoder gives some helpful hints on how to implement secure websockets using an HTTPS-capable nodejitsu instance.

Test Manifest dot com

gkoberger’s latest project testmanifest.com generates random open web app manifests for testing. Why would you need that? If you’re developing against tools that handle various different open web apps, then both Firefox and the Marketplace app will require these files be distinct. So potentially, you’d be spending a lot of time generating new, fake, web app manifests. Test Manifest helps you do exactly that: Edit and host a bunch of manifests to use for developing open web apps-related tools, without manually generating new ones all the time.

AppCloudy

gkoberger’s next, fantastic project is a dead-simple way to deploy open web apps that integrates with github. It takes your project’s source code, generates a manifest and appcache file, compiles LESS and coffeescript files, and much more! And if that wasn’t enough yet, it also works on both the Mozilla Marketplace and the Chrome Web Store. Check it out at appcloudy.com.

SmartSync for Shipyard JS

seanmonstar is working on a new feature for his Shipyard JS JavaScript framework: SmartSync. It enables the developer to provide a backup storage provider for when the browser is offline rather than just failing to interact with the server and making the application useless until network connection is restored. This will be particularly useful in mobile environments, enabling the user to make local changes to their application, which will be buffered until the device goes back online, at which time the actions will automatically be uploaded to the server. The feature is still developed in a feature branch, but is expected to land in Shipyard’s main code base soon.

Status of Project Petri

peterbe gives a quick update on the status of Mozilla’s Project Petri, Platform-as-a-Service and Infrastructure-as-a-Service for Mozilla developers, based on CloudFoundry (“VMWare-equivalent of Heroku”) and Eucalyptus (an open Amazon EC2 clone), respectively. They are facing some unique challenges deploying Playdoh-based Django projects to CloudFoundry.

ArtMaker

Also known as Drawing for people who can’t draw, lorchard is working on a little collage maker written fully in JavaScript. You can stack various cliparts on top of each other and manipulate them, and finally export the result as a PNG file or send it straight to imgur. The tool is particularly useful for artistically challenged people who’d like to generate badges, or for making a logo for your add-on or open web app. Les has a demo up on his domain and would love you to fork this on github!

The video

If you’d rather watch it for yourself, here you go:

We hope you enjoyed this month’s Beer and Tell and will join us again in June! If you have any questions, remarks, or concerns, don’t hesitate to comment on this post!

At PyCon 2011, Brian Moloney (of Imaginary Landscape here in Chicago) gave a talk titled “How to Get A Python Job”. It wasn’t really so much about getting specifically a Python job as about how to land any job. It was the same sort of advice you can find in a lot places – hints on how (not) to apply, how (not) to present yourself, how (not) to interview, how (not) to impress interviewers, and so on. If you’re looking for a job, go watch it and take his advice very seriously.

I went to Brian’s talk because I had just found my own Python job and I was curious about his perspective. I had started my own job search with my fair share of negatives – the economy was in the toilet, I was over 50, I didn’t have much formal training, my experience (in private education) wasn’t exactly mainstream, etc. And yet within 3 months of sending out my first resumé I had a job, and even had to turn down other offers. Even as I settled into my new job, I was frequently contacted by recruiters looking for Python programming help. And yet, I’ve heard many people complaining quite vocally about how difficult it was (and is) to find programming jobs in general and Python jobs in particular.

As the IT director and lead developer at a startup trying to grow its tech staff, I’ve been heavily involved in searching for and interviewing developers. In that process I’ve seen people of all ages and backgrounds painfully ignore the basics of getting a job. So the idea behind this post is to put one more voice out there, sharing some of the things that make me not want to hire an applicant.

Here are the things that make crazy. Take from them what you will.

Not knowing the market. Several candidates have said, “it seems like no one is looking for Python people.” Say what? I get called regularly by recruiters looking for Python developers. We’ve been searching for a Python dev for months. Admittedly, the market is better for more experienced people and you might need to relocate, but even junior Python dev positions are going begging. Ask any recruiter, we’re a hot commodity. When someone tells me that no one is hiring for Python, that tells me one of two things… either that person doesn’t know the job market (at least around here), or they aren’t really a Python developer. In the first case, I’m a pretty firm believer that the responsibility for finding jobs lies with the job seeker and not having done research on the market it a bad sign. If you don’t care enough to do research to find your own career, why should I expect that you’ll have the initiative to solve problems and help our team move forward? And the alternative is worse – in a market where knowing how to spell the word “Python” (at least on a resumé) should get you at least a few calls, not getting any traction is a red flag.

Not knowing the craft. This is so obvious that it shouldn’t need saying… except for the fact that it needs saying. “If you’re applying for a Python programming job, you should know as much as you can about Python and programming.” That means a basic understanding of the language, some basic knowledge of programming, some familiarity with the common or hot tools and technologies, etc. Obviously the depth of that understanding will vary depending on your experience, and for junior positions, particularly, you’re not expected to know everything, but you still should have some idea of the field.

I’ve interviewed candidates who couldn’t tell me how they would approach a problem, or what libraries they would use (or even have used), who’ve never even heard of Django, and so on. I’m no fan of springing little programming puzzles on candidates, since to my mind that mostly tests the ability to solve little puzzles, and I don’t expect lengthy dissertations on advanced programming topics, but I do expect you to be able to know the basics of coding and Python syntax, to have at least a nodding familiarity with current technologies, and to be able to discuss the approaches, tools and pitfalls involved in some of the coding you’ve done.

And speaking of the coding you’ve done, particularly for junior developers, you may not have done that much professionally. However, if you haven’t done anything outside of classwork, I’m going to be pretty skeptical. There are tons of opportunities to get involved in open source projects, to create your own web site, to help with something for a business, a club, or even just scratch your own personal software “itch.” It doesn’t matter how big or how successful your projects were, what matters to me is that you’re interested enough to be in the game on your own time.

Not knowing us (and the job). It’s been said before, but if you don’t care enough to do some research and thinking about us and the position before you come in for an interview, why should we think you’ll care once you’re hired? You’ve got the job description, probably some names of people at the company, and the entire Internet as resources. Even if you don’t have absolutely all the information, you should be able to find out enough to ask questions and maybe even float trial solutions. Why do you do x? How do you deal with issue y? Have you ever thought of trying z to solve this problem? Even if you’re completely wrong about your assumptions, the very fact that you’re thinking about our business is very compelling. If you want to be really sneaky talk about how “we” might solve some of those problems together.

And speaking of sneaky (not really, since it’s clear if you’ve done it or not), if you know who you’ll be interviewing with, look the people up online. Google and LinkedIn are your friends here. At worst you’ll have a feeling of control during the interview and at best you can work in a comment or question relevant to the interviewer’s background, which is almost always good. I’m not going to support hiring you just because you looked at my LinkedIN profile, but it does tell me you cared enough to do some research, and that’s a good thing.

Not selling yourself. Don’t go into your shell – there is a job open and somehow you’ve gotten an interview. So take advantage of it! You may be an introvert, it may be hard to talk about yourself, or maybe you just don’t think you should have to “market” yourself. Whatever… but in this situation you’ve really got to go for it. Show us how interesting you are, how much you care about the work, how cool you think the job is, how much you would bring to the team. If we hire you, we’ll have to work with you every day, so why should we look forward to that? (In a business sense, of course – promising to keep everyone loose with dirty jokes is probably not be the way to go here.)

We can only hire a few people and we need to get a lot of stuff done. We have lots of problems crying out for solutions so we want someone who is eager to tackle those problems and has some chance of succeeding. As Joel Spolsky succently put it, “smart and gets things done.” You may not have the exact sklls to fix every problem at this moment, but that’s not the entire point – as a startup none of us had the exact skillset for this business when we started. But all of us were ready to dig in, we figured it out fast, and we got the job done. And that’s the sort of person we need.

Note: this is not the same as emphasizing how good this job would be for you. I understand that you want to move from that old job to a new, more interesting area. I applaud the fact that you just love to learn new things. I’m totally on board with how wonderful this opportunity would be for you… but we need to move and scrub a ton of data, polish the UI, analyze customer behavior, and keep thinking of new ways to stay ahead of the game. How are you going to help us do that?

Now I have a dare for you. Let me end this lengthy rant with a challenge – if you’re a junior Python developer willing to work in the Chicago area, I dare you to send us your resumé. We’re Zoro Tools, Inc and you can send it to either Bruce Simmons, our recruiting lead (bruce.simmons<AT>zorotools.com), or me (vernon.ceder<AT>zorotools.com). If you take all of the above to heart we’ll be bowled over and have no choice but to hire you. That’ll serve me right for pontificating on hiring.

[edited to add company name and contact info]

Since then, I've done more work as maintenance programmer on a Django project, and I've been reminded that library and framework design must take into account the fact that not all developers are experts. Even if you only hire the best, no-one can be an expert straight away.

Thinking through things more from the perspective of a maintenance programmer, my doubts about CBVs have increased, to the point where I recently tweeted that CBVs were a mistake.

So I thought I'd explain my reasons here. First, I'll look at the motivation behind CBVs, how they are doing at solving what they are supposed to solve, and then analyse the problems with them in terms of the Zen of Python.

What problems do CBVs solve?

Customising generic views

People kept wanting more functionality and more keyword arguments to the list_detail views (and others, but that those especially, as I remember). The alternative was large copy-and-paste of the code, so people were understandably wanting to avoid that (and avoid writing any code themselves).

So, we replaced them with classes that allows people to override just the bit they need to override. This eliminates the need for code duplication, and removes the burden of lots of feature requests for generic views.

Or does it? Instead of tickets for keyword arguments to list_detail, it seems we have a bunch of other tickets asking for changes to CBVs, many of which can't be implemented as mixins or subclasses.

One of the problems is that if the view calls anything else (e.g. a paginator class, or a form class), you have to provide hooks for how it calls it, which means implementing methods that can be overridden. If you forget any, or if the thing you are calling gains some new keyword arguments, you've got feature requests, or duplication because someone had to override a larger method just to change one aspect of it. If you don't forget any, you've got dozens of little methods to document.

Also, there are problems like this attempt to mix FormView with ListView functionality. Fixing this will end up with similar amounts of copy-paste, but in this case it requires a fair bit of debugging first to realise you have a problem.

So I'm not convinced CBVs have made much difference here.

Eliminating flow control boilerplate

The classic example is editing using a form. You see this pattern again and again using function based views (FBVs from now on):

from django.shortcuts import render

def contact(request):
    if request.method == 'POST':
        form = ContactForm(request.POST)
        if form.is_valid():
            send_contact_message(form.cleaned_data['email'],
                                 form.cleaned_data['message'])
            return HttpResponseRedirect('/thanks/')
    else:
        form = ContactForm()

    return render(request, 'contact.html', {'form': form})

Without question this is tedious and annoying. CBVs reduce this to:

from django.views.generic.edit import ProcessFormView

class ContactView(ProcessFormView):
    form_class = ContactForm
    template_name = 'contact.html'
    success_url = '/thanks/'

    def form_valid(self, form):
        send_contact_message(form.cleaned_data['email'],
                             form.cleaned_data['message'])
        return super(ContactView, self).form_valid(form)

Much better!

However...

It's not really that much shorter. 8 lines compared to 11, ignoring imports. But now let's make it more realistic. We're going to have:

• Initial arguments to the form that are based on the request object.
• Priority users get a form with the option to indicate 'urgent' status to message, which results in a text message as well as an email. The template is also rendered a bit differently for them and needs a flag.
• URLs defined using reverse as they should be.

FBV:

from django.core.urlresolvers import reverse
from django.shortcuts import render

def contact(request):
    high_priority_user = (not request.user.is_anonymous()
                          and request.user.get_profile().high_priority)
    form_class = HighPriorityContactForm if high_priority_user else ContactForm

    if request.method == 'POST':
        form = form_class(request.POST)
        if form.is_valid():
            email, message = form.cleaned_data['email'], form.cleaned_data['message']
            send_contact_message(email, message)
            if high_priority_user and form.cleaned_data['urgent']:
                 send_text_message(email, message)
            return HttpResponseRedirect(reverse('contact_thanks'))
    else:
        form = form_class(initial={'email': request.user.email}
                                  if not request.user.is_anonymous() else {})

    return render(request, 'contact.html', {'form': form,
                                            'high_priority_user': high_priority_user})

CBV:

from django.core.urlresolvers import reverse_lazy
from django.views.generic.edit import ProcessFormView

class ContactView(ProcessFormView):
    template_name = 'contact.html'
    success_url = reverse_lazy('contact_thanks')

    def dispatch(self, request, *args, **kwargs):
        self.high_priority_user = (not request.user.is_anonymous()
                                   and request.user.get_profile().high_priority)
        return super(ContactView, self).dispatch(request, *args, **kwargs)

    def get_form_class(self):
        return HighPriorityContactForm if self.high_priority_user else ContactForm

    def get_initial(self):
        initial = super(ContactView, self).get_initial()
        if not request.user.is_anonymous():
            initial['email'] = request.user.email
        return initial

    def form_valid(self, form):
        email, message = form.cleaned_data['email'], form.cleaned_data['message']
        send_contact_message(email, message)
        if self.high_priority_user and form.cleaned_data['urgent']:
             send_text_message(email, message)
        return super(ContactView, self).form_valid(form)

    def get_context_data(self, **kwargs):
        context = super(ContactView, self).get_context_data(**kwargs)
        context['high_priority_user'] = self.high_priority_user
        return context

(A few lines could be shaved by not using super() in a number of places, but only at the expense of future confusion/problems if a maintainer was expecting the normal declarative behaviour.)

Notice:

1. I really want high_priority_user to be a local variable that is calculated once, and used in a couple of places. With a function, that's what I have. With a CBV, I have to simulate it using an attribute on self. I also need to override the dispatch() method just to create it. These are both ugly hacks.

2. The CBV version is extremely noisy, due to all the calls to super(). This would be improved in Python 3 (at the expense of a rather magical super() builtin), but is still far from perfect. Every time you override a method, you have to mention it twice.

3. The CBV version is now significantly longer - 24 non-blank lines compared to 17. Since I'm using the same APIs for requests and forms, and doing the same thing, this can only mean that the amount of boilerplate has significantly increased. Sure, I've removed the flow control boilerplate, but must have added some other type.

4. Even if you know the Form API very well, you are going to have to look up the docs or the source code to find get_initial() etc. and ensure you get the signature correct. You have to know two sets of APIs to use a form, instead of one.

5. In the CBV, flow control is totally hidden. In the process of abstracting away the duplication, we've also hidden the order of execution. I've ordered my methods in the order they are called (I think), but that may or may not be obvious to anyone else, and nothing forced me to do it.

6. Because of the last point, it is massively more difficult to debug. Which of the two would you rather maintain? Which do you think a maintenance programmer, who has never seen this code before, would rather maintain?

   To understand what is going on, you've got an intimidating stack of base classes to navigate, compared to a single function.

The fundamental problem here is that Python sucks at implementing custom flow control. Not many languages shine here. Ruby has blocks, which help. Haskell has a pretty good story due to a combination of succinct function definition, lazy evaluation and the way that IO works. Lisp has macros. But with Python, we are limited to: abusing generators, abusing the with statement, or classes and the template method design pattern (which is basically what CBVs use).

I think we decided to use the latter because it's one of the only options we've got, but failed to notice that it's just not very good.

There are worse things that can happen with shifting requirements. What if you want two different forms on the same page? I've done this on more than one occasion, and it can be a useful thing — when you present the user with two different courses of action, and you've got completely different info they need to fill in.

It would be a nightmare to get CreateView or UpdateView to do this. You will either produce a monstrosity, or you'll have to start from scratch with an FBV. You've been seduced down a tempting, calm stretch of water, and then left high and dry when you come to the end of what CBVs offer. If you start with an FBV, it's an easy change.

Overall, when I work on CBVs, even views I've created, I start to feel like I'm working on a classic ASP.NET Page class. It is bringing back painful memories! We’re not as bad as that yet, but methods that simply modify some data on self are a code-smell that we are heading in that direction.

Another way of looking at it is that with CBVs, views have become an instance of the ‘framework pattern’ instead of the ‘library pattern’. With a framework, your application code gets called by the framework code, and you can easily end up having to understand how the framework is implemented.

With a library, the library code gets called by your application code, and this is in general much more flexible, much easier to document and much easier to debug. We ought to be moving Django more in the direction of a library.

With the comparison to ASP.NET, I'm also basically saying CBVs are not Pythonic. I'll back up this claim in terms of selected parts of the Zen of Python.

Are CBVs Pythonic?

Beautiful is better than ugly

This is a subjective one, but the noise of all the super() calls, get_context_data() compared to a simple {} etc. is ugly to me.

Simple is better than complex

Let's notice, first off, that FBVs are simpler than CBVs, according to the basic meaning of having fewer parts.

A class is a datastructure with attributes and functions attached. A function is just a function. So a class is more complex than a function. If we ever use a class where a function would work, we have to justify the additional complexity.

Complex is better than complicated

Are CBVs complicated? Just read the Django source if you think they are not.

In one app I wrote, I had a mixin that implemented a bit of common flow control for forms — namely it returned a JSON response with validation errors for certain types of requests. This meant I didn't need separate views for the AJAX validation, and with CBVs I eliminated all the boilerplate. Great!

Well, it was, until I found a crazy problem to do with the order in which different base classes set things up. To solve my problem, I ended up writing this:

# MRO problem: we need BaseCreateView.post to come first in MRO, to
# provide self.object = None, then AjaxyFormMixin must be called before
# ProcessFormView, so that the right thing happens for AJAX.

class AjaxMroFixer(type):

    def mro(cls):
        classes = type.mro(cls)
        # Move AjaxyFormMixin to one before last that has a 'post' defined.
        new_list = [c for c in classes if c is not AjaxyFormMixin]
        have_post = [c for c in new_list if 'post' in c.__dict__]
        last = have_post[-1]
        new_list.insert(new_list.index(last), AjaxyFormMixin)
        return new_list

It's a metaclass, and implements the mro() method that allows you to override the method order resolution. I am not proud of this code. (For those who don't get English understatement, please understand what I'm saying: this code is horrific). Before writing this code, I didn't know that the mro() method existed. Although I value the education, if a framework forces you to learn about type.mro() and implement it, it is doing something wrong.

Flat is better than nested

The hierarchy of classes is certainly a form of nesting, whereas functions are flat. I think CBVs might be considerably better if you started by writing your own, so that you had base classes that did everything you needed, but nothing more, and ended up with a very flat hierarchy.

Readability counts

Get some people who don't know Django to read ContactView above and figure out what it does, and some others to feed the function version, and see who has the easier time. There is no contest here.

Explicit is better than implicit

With a CBV, by inheriting from a class you are inheriting all the behaviour of that class, and all its parent classes. You could argue this is explicit, since you've explicitly indicated the base class, but you haven't indicated all the parent classes — they come automatically. So it's more like an implicit request in practice.

Of course, this is always true with OOP to some extent — you are inheriting a bunch of behaviour precisely because you don't want to define it again. But let's notice that it does have its downsides.

For example, let's play spot the difference between the following two views — an FBV:

from django.shortcuts import render

def my_view(request):
    return render(request, "my_template.html", {})

and a CBV:

from django.views.generic import TemplateView

class MyView(TemplateView):
    template_name = "my_template.html"

Can you see the important difference? Try to answer before reading on.

I'm talking only about the functional differences when this view is accessed by a client, not differences of code organisation or re-use or performance.

Well, TemplateView inherits from View which provides a dispatch() method, and TemplateView provides a get() method which will handle all GET (and HEAD) requests, by the logic defined in View.dispatch(). However, neither defines a post() method, which means that you will get a "405 Method Not Allowed" error if you try POST or other HTTP verbs, to the CBV view, whereas you will get a 200 with the FBV.

In the CBV, all of this logic has been invoked implicitly. Nothing in what I wrote was an explicit request for 405s for POST requests, but I got it because I inherited from TemplateView — even though using a template does not imply that behaviour.

(By the way, this issue caused a real problem in a site I wrote, which was easily debugged because I knew a lot about the internals of CBVs, and therefore easily fixed, but it still adds noise to my class — code that can only be explained by reference to some inherited behaviour I didn't really want).

Of course, as already mentioned, you can say the same thing whenever you inherit from a class. But, in my opinion, it is one of the disadvantages of OOP, and it is showing itself here. The question is: do the advantages of inherited behaviour outweigh the disadvantages of implicit behaviour?

There should be one way to do it

I already mentioned one case where the CBV method is just not going to work for you, or is not going to be worth it — needing two different forms on the same page. But in the real views I write, I suspect probably the majority would not fit easily into a CBV.

So, in your project you now need both FBVs and CBVs — you've got two ways to do it. When a maintenance programmer comes along, and needs to do some similar work that involves a form, they will be confused. Which pattern do they start from?

The simple way to avoid this is to avoid the less flexible solution — avoid using CBVs.

So simply having two different ways of building up views is a violation of this principle, but within CBVs there are also instances.

First, there are also problems with the attempt to use declarative style, which is perhaps the most attractive feature of the CBV API. So, you need initial arguments to your form? Just define the initial attribute on your class. Unless, however, you need it to be dynamic — you'll have to define get_initial() instead.

Also, it often isn't obvious where to add certain bits of code. There is often more than one choice of method to override when you come to add a little bit of functionality e.g. get() or dispatch(), and which you choose sometimes has subtle implications, and sometimes doesn't matter. With FBVs, two different Django developers tasked with the same modification to an existing view would often produce identical or nearly identical patches, but I suspect that would be rarer with CBVs.

If the implementation is hard to explain, it's a bad idea

Looking at the implementation, you'll find things like MultipleObjectTemplateResponseMixin, as well as MultipleObjectMixin and TemplateResponseMixin, and the former is not just the composition of the other two. This is just one of many signs that things are going wrong. If you can really compose behaviour just by adding mixins, MultipleObjectTemplateResponseMixin should not be needed. Explaining things like this is hard, and the reason is that you just can't build up complex views using classes and mixins.

Conclusion

Overall, I think CBVs make:

• very simple views slightly shorter, much cleaner (and significantly harder to debug, but you don't need to debug them because they are simple);

• views of medium complexity significantly longer, with more boilerplate and noise and much harder to debug;

• views of high complexity almost impossible.

You only gain for the simple views, but they were simple anyway, just slightly tedious. Is this advantage really enough to outweigh the disadvantages I've listed?

To be honest, I regret dropping the function based generic views for CBVs. There was an alternative solution to the tickets asking for them to do more: WONTFIX.

Generic views were simple shortcuts for common problems. They didn't actually involve very much code, and if you needed to duplicate some of it you weren't duplicating much. We should have just said: this is what generic views do, if you need them to do something else then write your own. Because that is what you have to say with class based generic views anyway, just slightly later.

Regarding going forward, I say: stop the rot.

If you're starting a new project, I recommend avoiding CBVs, or just wrapping them in thin functional wrappers, like this one for ListView (or, with django-pagination a simple 3-line view function is probably all you need and is actually easier than subclassing ListView).

For Django core, let's fix up the main problems and bugs CBVs have, and then leave them as solutions to simple problems.

But please: let's not move everything in Django-world — whether Django core/contrib or resuable apps — to CBVs. Just use a function, and stop writing classes.

• "Blameless PostMortems and a Just Culture" -- John Allspaw describes the post mortem process at Etsy, in particular how it encourages engineers to own up to their mistakes and to help others avoid them, all in a safe and blame-free environment
• "Sprint.ly's Continuous Integration" -- good overview of what should be by now industry-standard methods and tools for rapid deployments and continuous integration
• "Big List of 20 Common Bottlenecks" -- via the super useful High Scalability blog, a list of bottlenecks that you will hit one way or another if you do any serious high-volume systems work
• "BASE: An ACID alternative" -- Dan Pritchett from EBay coined the term BASE (basically available, soft state, eventually consistent) to compare and contrast NoSQL systems with traditional ACID-based relational databases; very good overview of these types of systems
• "Calvin: Fast Distributed Transactions for Partitioned Database Systems" (PDF) -- Daniel Abadi and collaborators write about a distributed transaction mechanism that can sit on top of non-transactional storage systems, transforming them into scalable, highly-available ACID databases
• "Creating a BOSH from scratch on AWS" -- great tutorial from Dr Nic from Engine Yard on installing CloudFoundry's BOSH tool on AWS
• "Amazon's Journey to the Cloud" -- very good presentation by John Rauser on the long and winding road taken by Amazon from their beginnings running on 1 machine to the launch of AWS technologies
• "Engineering Change" -- short but very insightful presentation by Etsy's CTO Kellan Elliott-McCrea on continuous deployment strategies and metrics-driven development
• "People Make Poor Monitors for Computers" -- eye-opening article on the dangers on relying on highly automated and sophisticated monitoring systems; when they fail, human operators are expected to jump in and fix the issues, but unfortunately those rare issues are extremely hard to diagnose and fix by humans who have lost their edge by relying on the automated systems in the first place!
• "vbench - benchmarking performance through time" -- from Wes McKinney's Panda project, vbench is a lightweight Python library for measuring code performance and catching performance regressions
• "Big Data -- a little analysis" -- switching gears to Big Data, here's a good overview/taxonomy of types of problems in this space, based on data volume and algorithm complexity, courtesy of Chris Swan
• "The unsexy side of big data: 5 tools to manage your Hadoop cluster" -- some tools I had never heard of for managing Hadoop clusters, including Apache Ambari and Apache Mesos
• "Online resources for handling big data and parallel computing in R" -- from the R-bloggers blog aggregator, a useful collection of links on mostly parallel computing with R
• "Much to like about HBaseCon" -- quick overview of some talks from last week's HBaseCon 2012

The Chromium WebAudio demos don't work yet in Epiphany because WebKitGTK+ doesn't yet ship with WebAudio support enabled by default and the WebView used by Epiphany doesn't yet enable the enable-web-audio WebKitWebSetting. However, for the brave people willing to build WebKitGTK+, here are the steps to follow:

$ Tools/Scripts/build-webkit --gtk --web-audio
$ Tools/Scripts/run-launcher --gtk --enable-web-audio=TRUE http://chromium.googlecode.com/svn/trunk/samples/audio/shiny-drum-machine.html

I especially like the drum machine demo, pretty impressing stuff! So, back to the topic. Fortunately the WebAudio AudioContext was designed to allow multiple platform backends in WebCore, I'm going to talk about the GStreamer backend. It was first prototyped by Zan Dobersek and he kindly passed me on the task which I continued and upstreamed in WebKit trunk.

The two biggest features of the backend are:

1. Decoding audio data from a media file or from a memory pointer and pulling it into the WebAudio machinery of WebKit where it's going to be processed by the WebAudio pipeline.

2. Getting back audio data as WAV from the WebAudio pipeline and play it back :)

For the first step we use decodebin2 with data coming from either filesrc or giostreamsrc. The decoded data is split in mono channels using the deinterleave element, passed to appsinks and converted to AudioChannels which internally store the data in FloatArrays. Every AudioChannel is stored in an AudioBus instance which we pass onto WebCore for further internal processing. The implementation can be found in the AudioFileReaderGStreamer.cpp file.

Internally the WebAudio stack also relies on our GStreamer FFTFrame implementation which uses the gst-fft library to perform Fast Fourier Transforms.

Then let's talk about playback! Once the WebAudio stack has finished processing data and if the web developer created an AudioDestinationNode as part of his WebAudio pipeline, the user needs to hear something playing :) As the mighty reader might guess the magic is done in an AudioDestinationGStreamer.cpp file! I decided to implement most of the logic about reading data from the AudioBus as a GStreamer source element, called WebKitWebAudioSourceGStreamer. This element takes every AudioChannel of the AudioBus, convert the FloatArray data to GStreamer buffers, interleave the channels and encode the whole as WAV data using the wavenc element. In the AudioDestination GStreamer pipeline we then use this shiny source element, parse the WAV data and pass it on the audio sink!

What's next in our roadmap?

• I'm not sure this source element was the correct approach in the end, I think that at some point I'll just refactor it into the AudioDestination implementation.
• Port to GStreamer 0.11 APIs! This work is on-going already for the MediaPlayer.
• There are still some bugs to fix, before I really feel confident about enabling this feature by default. We should also add support for reading data from the HTML5 Media elements.
• The WebKit webaudio layout tests are almost passing on the build bots, this work is on-going as well.
• Enable WebAudio by default in WebKitGTK's build and Epiphany, targetting GNOME 3.6.

I would like to thank Igalia for allowing me to dedicate work time on this project :)

Obviously, this simplistic view of the world fails to take into account the constant battle with the U.S. Fish and Wildlife Service in the Everglades but most of all, it fails to take into account that the Python language itself is a very small contribution compared to the tools and the body of knowledge shared within the community.

Of course, we all understand that development is more than just writing code. On June 12th, Montréal-Python is going to celebrate our unwritten lore with Nootropic Nutrients, a meeting on the Best Practices and tools in the Python world.

Can you claim a ten-fold speedup compared to developement with traditional languages? Do you have hard-won secrets that you would like to share? Come and tell us about your tools and stategies for efficiently and consistently delivering your projects on time.

We are looking for speakers for presentations ranging from 5 minutes to 40 minutes.
If you are interested, send us an email to mtlpyteam@googlegroups.com with a short description of your talk.

The meeting will follow the usual schedule, starting around 6:30pm and moving to the pub for the follow-up discussions around 8:30pm.

Last week's Kotti sprint in Berlin was full of win. Here's a summary of what we worked on. (Sprinters: If I forgot to mention anything, please let me know and I'll update.)

(Update: Saving a page on the demo site seems to be broken as we speak. Will fix soon.)

Snippets, slots and pyramid_snippets

Christian, Florian, Krille and Ibi worked on adding support for what Wordpress calls Shortcodes and MoinMoin calls macros.

With this, users will be able to insert codes like [gallery] into their pages and have these render to dynamic elements in the public view. Shortcodes may also take arguments like [table-of-contents level=3]. We discussed this idea back in February on the list.

The team decided they would move the core functionality behind what they call snippets into a separate package pyramid_snippets. The idea is that snippets behave like views with the advantage that they can be easily pulled in via AJAX and be used in other places in the site, like in the portlet columns. The integration with Kotti is happening in the snippets branch. Plans are to add a button to TinyMCE that allows users to add snippets instead of having to remember the codes and syntax.

We also want to eventually deprecate Kotti's slots and unify them with snippets so that slot renderers will become views and can be used as snippets and vice versa.

Paster templates

Marco and Tom worked on kotti_paster which Marco initially started. They've been working on a set of accumulative templates for Kotti that are based around Buildout and try to define best practices for different aspects of Kotti and Pyramid development for beginners.

Questioner: i have a problem with a little recursive function
Questioner: i create a list recursive of length 2**30 but i get a memory problem
Naysayer: recursion in Python is almost always bad news
Questioner: i have probs with high memory because i create too many too large list when i enter a value of 2**30 for N
RecursionFan: recursion is awesome
RecursionFan: it sounds like you're doing it wrong
Naysayer: you should avoid recursion in python
RecursionFan: why?
Naysayer: There's this little thing called the stack
Questioner: yeah i now the depth of my recursion, its just log(N)
Questioner: but can somebody help me so this function can work for parameters of 2**30
RecursionFan: there's this little thing called http://en.wikipedia.org/wiki/Tail_call
Naysayer: yeah, and python doesn't have it
DeadPanda: RecursionFan, does python optimise tail-recursion?
nedbat: RecursionFan: Python doesn't do tail-call removal.
Naysayer: DeadPanda: No
nedbat: Naysayer: "you should avoid recursion" is a bit strong. You need to understand the limitation.
nedbat: Naysayer: Python is as good at recursion as C is at integer arithmetic.
Naysayer: nedbat: It is good practice to avoid recursion in Python if you don't understand how it works.
nedbat: Naysayer: it is good practice to avoid _____ in Python if you don't understand how it works.

(Names other than mine, nedbat, are changed.) Before anyone's read Questioner's code, or even listened fully to his question, we're off on the usual rant against recursion in Python, and debates about tail-call elimination. As it happens, and as Questioner rightly understood, the problem is not about deep recursion:

def computeHW(N):
    if N == 1:
        return [1]
    else:
        L = computeHW(N/2)
        L.extend([2*i for i in L])
        return L

Questioner wanted to call this function with 2**30, which is quite modest in terms of depth, the stack will only get 30 levels deep. But it's crazy in memory terms: it tries to allocate a list with a billion elements, but not before it's allocated and deallocated many many others, up to that size.

So yes, this function isn't going to work, but not because of tail-call optimization or its lack. We can change the function into a generator that still works recursively, but won't allocate any memory:

def computeHW2(N):
    if N == 1:
        yield 1
    else:
        for i in computeHW2(N/2):
            yield i
        for i in computeHW2(N/2):
            yield 2 * i

Talking more with Questioner, it turns out he didn't want the sequence, he wanted to be able to randomly access elements of his list. We can turn this function into a closed-form solution that computes the n'th element of this list:

def closedHW(N):
    if N == 0:
        return 1
    else:
        bits = N.bit_length()
        return 2 * closedHW(N - 2**(bits-1))

This function is still recursive, and so will use up to 30 stack frames for numbers up to 2**30, but doesn't allocate any lists at all. This function is also in a simple tail-call form, so we can easily make it iterative:

def closedHW_iterative(N):
    hw = 1
    while N:
        bits = N.bit_length()
        N -= 2**(bits-1)
        hw *= 2
    return hw

BTW, this code gets to use one of the few methods on integers, bit_length(), which is new in 2.7.

In the end, we eliminated the recursion, but not for any of the reasons Naysayer was darkly hinting at.

When people say Python is bad at recursion, they are referring to the fact that the stack can't grow beyond 1000 frames. This makes it bad at the kind of recursion that languages like Scheme and Haskell encourage. This school of thought uses recursive procedures for iterative processes, to put it in terms from Structure and Interpretation of Computer Programs. Python is not good for this sort of recursion, because it won't work for more than 1000 iterations, which is far too low a limit.

But truly recursive processes don't often need that much stack, so using recursive procedures for recursive processes is fine in Python. Blanket statemtents like "recursion is almost always bad new in Python" are just simplistic.

• coloring raw text data inside editor on load
• coloring text data when editor is cloned
• updating document highlight on line edits
• providing color palette (scheme) for the editor
• providing data for Outliner

• background highlight for current line
• background highlight for search / current line occurrences

```python
print "hello world"
```

will render as:

<pre><code class="python">
print "hello world"
</code></pre>

The advantages are that you don't have to indent the code (which is useful when you copy and paste python code) and that you can add a language identifier which can be used to highlight the code with tools like highlight.js.

Adding github flavored markdown code blocks to python markdown is really easy. There is already the fenced code blocks extension. All you have to do is to customize the regular expression in markdown/extensions/fenced_code.py:

67c67
<     r'(?P<fence>^~{3,})[ ]*(\{?\.(?P<lang>[a-zA-Z0-9_-]*)\}?)?[ ]*\n(?P<code>.*?)(?P=fence)[ ]*$', 
---
>     r'(?P<fence>^`{3,})[ ]*(?P<lang>[a-zA-Z0-9_-]*)[ ]*\n(?P<code>.*?)(?P=fence)[ ]*$', 

What do you think this script prints?:

If you guessed something like this:

…then you’d be right, in Python 2.7 and 3.x. In Python 2.6 and older, the order of messages is reversed:

In New Python, the Weeper is deleted as soon as its thread dies, and __del__ runs on the dying thread. In Old Python, the Weeper isn’t deleted until the thread is dead and a different thread accesses the local’s __dict__. Thus the Weeper is deleted at the line getattr(local, 'whatever', None), after the thread dies, and Weeper.__del__ runs on the main thread.

What if we remove the getattr call? In Old Python, this happens:

Without getattr, the Weeper isn’t deleted until interpreter shutdown. The shutdown sequence is complex and hard to predict—in this case the thread module has been set to None before the Weeper is deleted, so Weeper.__del__ can’t do thread.get_ident().

Thread Locals in Old Python

To understand why locals act this way in Python 2.6 and older, let’s look at the implementation in C. The core interpreter’s PyThreadState struct has a dict attribute, and each threading.local object has a key attribute formatted like "thread.local.<memory address of self>". Each local has a __dict__ of attributes per thread, stored in PyThreadState‘s dict with the local’s key.

threadmodule.c includes a function _ldict(localobject *self) which takes a local and finds its __dict__ for the current thread. _ldict()finds and returns the local’s __dict__ for this thread, and stores it in self->dict.

This architecture has, in my opinion, a bug. Here’s the implementation of _ldict():

I’ve edited for brevity. There’s a few interesting things here—one is the check for a custom __init__ method. If this object is a subclass of local which overrides __init__, then __init__ is called whenever a new thread accesses this local’s attributes for the first time.

But the main thing I’m showing you is the two calls to Py_CLEAR(self->dict), which decrements self->dict‘s refcount. It’s called when a thread accesses this local’s attributes for the first time, or if this thread is accessing the local’s attributes after a different thread has accessed them—that is, if self->dict != ldict.

So now we clearly understand why a thread’s locals aren’t deleted immediately after it dies:

1. The worker thread stores a Weeper in local.weeper. _ldict() creates a new __dict__ for this thread and stores it as a value in PyThreadState->dict, and stores it in local->dict. So there are two references to this thread’s __dict__: one from PyThreadState, one from local.
2. The worker thread dies, and the interpreter deletes its PyThreadState. Now there’s one reference to the dead thread’s __dict__: local->dict.
3. Finally, we do getattr(local, 'whatever', None) from the main thread. In _ldict(), self->dict != ldict, so self->dict is dereferenced and replaced with the main thread’s __dict__. Now the dead thread’s __dict__ has finally been completely dereferenced, and the Weeper is deleted.

The bug is that _ldict() both returns the local’s __dict__ for the current thread, and stores a reference to it. This is why the __dict__ isn’t deleted as soon as its thread dies: there’s a useless but persistent reference to the __dict__ until another thread comes along and clears it.

Thread Locals in New Python

In Python 2.7 and 3.x, the architecture’s a little more complex. Each PyThreadState‘s dict contains a dummy for each local, and each local holds a dict mapping weak references of dummies to a per-thread __dict__.

When a thread is dying and its PyThreadState is deleted, weakref callbacks fire immediately on that thread, removing the thread’s __dict__ for each local. Conversely, when a local is deleted, it removes its dummy from PyThreadState->dict.

_ldict() in New Python acts more sanely than in Old Python. It finds the current thread’s dummy in the PyThreadState, and gets the __dict__ for this thread from the dummy. But unlike in Old Python, it doesn’t store a extra reference to __dict__ anywhere. It simply returns it:

This whole weakrefs-to-dummies technique is, apparently, intended to deal with some cyclic garbage collection problem I don’t understand very well. I believe the real reason why Python 2.7 acts as expected when executing my script, and why Python 2.6 acts weird, is that 2.6 stores the extra useless reference to the __dict__ and 2.7 does not.

Kasvu Open Forum

Djangocon Finland 2012

Conclusion

Currently we have three versions of answers to the task in Python:

Python: function returning a function

>>> def fiblike(start):
 addnum = len(start)
 memo = start[:]
 def fibber(n):
  try:
   return memo[n]
  except IndexError:
   ans = sum(fibber(i) for i in range(n-addnum, n))
   memo.append(ans)
   return ans
 return fibber
 
>>> fibo = fiblike([1,1])
>>> [fibo(i) for i in range(10)]
[1, 1, 2, 3, 5, 8, 13, 21, 34, 55]
>>> lucas = fiblike([2,1])
>>> [lucas(i) for i in range(10)]
[2, 1, 3, 4, 7, 11, 18, 29, 47, 76]
>>> for n, name in zip(range(2,11), 'fibo tribo tetra penta hexa hepta octo nona deca'.split()) :
 fibber = fiblike([1] + [2**i for i in range(n-1)])
 print('n=%2i, %5snacci -> %s ...' % (n, name, ' '.join(str(fibber(i)) for i in range(15))))
 
 
n= 2,  fibonacci -> 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 ...
n= 3, tribonacci -> 1 1 2 4 7 13 24 44 81 149 274 504 927 1705 3136 ...
n= 4, tetranacci -> 1 1 2 4 8 15 29 56 108 208 401 773 1490 2872 5536 ...
n= 5, pentanacci -> 1 1 2 4 8 16 31 61 120 236 464 912 1793 3525 6930 ...
n= 6,  hexanacci -> 1 1 2 4 8 16 32 63 125 248 492 976 1936 3840 7617 ...
n= 7, heptanacci -> 1 1 2 4 8 16 32 64 127 253 504 1004 2000 3984 7936 ...
n= 8,  octonacci -> 1 1 2 4 8 16 32 64 128 255 509 1016 2028 4048 8080 ...
n= 9,  nonanacci -> 1 1 2 4 8 16 32 64 128 256 511 1021 2040 4076 8144 ...
n=10,  decanacci -> 1 1 2 4 8 16 32 64 128 256 512 1023 2045 4088 8172 ...
>>> 

Python: Callable class

>>> class Fiblike():
 def __init__(self, start):
  self.addnum = len(start)
  self.memo = start[:]
 def __call__(self, n):
  try:
   return self.memo[n]
  except IndexError:
   ans = sum(self(i) for i in range(n-self.addnum, n))
   self.memo.append(ans)
   return ans
 
 
>>> fibo = Fiblike([1,1])
>>> [fibo(i) for i in range(10)]
[1, 1, 2, 3, 5, 8, 13, 21, 34, 55]
>>> lucas = Fiblike([2,1])
>>> [lucas(i) for i in range(10)]
[2, 1, 3, 4, 7, 11, 18, 29, 47, 76]
>>> for n, name in zip(range(2,11), 'fibo tribo tetra penta hexa hepta octo nona deca'.split()) :
 fibber = Fiblike([1] + [2**i for i in range(n-1)])
 print('n=%2i, %5snacci -> %s ...' % (n, name, ' '.join(str(fibber(i)) for i in range(15))))
 
 
n= 2,  fibonacci -> 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 ...
n= 3, tribonacci -> 1 1 2 4 7 13 24 44 81 149 274 504 927 1705 3136 ...
n= 4, tetranacci -> 1 1 2 4 8 15 29 56 108 208 401 773 1490 2872 5536 ...
n= 5, pentanacci -> 1 1 2 4 8 16 31 61 120 236 464 912 1793 3525 6930 ...
n= 6,  hexanacci -> 1 1 2 4 8 16 32 63 125 248 492 976 1936 3840 7617 ...
n= 7, heptanacci -> 1 1 2 4 8 16 32 64 127 253 504 1004 2000 3984 7936 ...
n= 8,  octonacci -> 1 1 2 4 8 16 32 64 128 255 509 1016 2028 4048 8080 ...
n= 9,  nonanacci -> 1 1 2 4 8 16 32 64 128 256 511 1021 2040 4076 8144 ...
n=10,  decanacci -> 1 1 2 4 8 16 32 64 128 256 512 1023 2045 4088 8172 ...
>>> 

Generator

from itertools import islice, cycle
 
def fiblike(tail):
    for x in tail:
        yield x
    for i in cycle(xrange(len(tail))):
        tail[i] = x = sum(tail)
        yield x
 
fibo = fiblike([1, 1])
print list(islice(fibo, 10))
lucas = fiblike([2, 1])
print list(islice(lucas, 10))
 
suffixes = "fibo tribo tetra penta hexa hepta octo nona deca"
for n, name in zip(xrange(2, 11), suffixes.split()):
    fib = fiblike([1] + [2 ** i for i in xrange(n - 1)])
    items = list(islice(fib, 15))
    print "n=%2i, %5snacci -> %s ..." % (n, name, items)

[1, 1, 2, 3, 5, 8, 13, 21, 34, 55]
[2, 1, 3, 4, 7, 11, 18, 29, 47, 76]
n= 2,  fibonacci -> [1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610] ...
n= 3, tribonacci -> [1, 1, 2, 4, 7, 13, 24, 44, 81, 149, 274, 504, 927, 1705, 3136] ...
n= 4, tetranacci -> [1, 1, 2, 4, 8, 15, 29, 56, 108, 208, 401, 773, 1490, 2872, 5536] ...
n= 5, pentanacci -> [1, 1, 2, 4, 8, 16, 31, 61, 120, 236, 464, 912, 1793, 3525, 6930] ...
n= 6,  hexanacci -> [1, 1, 2, 4, 8, 16, 32, 63, 125, 248, 492, 976, 1936, 3840, 7617] ...
n= 7, heptanacci -> [1, 1, 2, 4, 8, 16, 32, 64, 127, 253, 504, 1004, 2000, 3984, 7936] ...
n= 8,  octonacci -> [1, 1, 2, 4, 8, 16, 32, 64, 128, 255, 509, 1016, 2028, 4048, 8080] ...
n= 9,  nonanacci -> [1, 1, 2, 4, 8, 16, 32, 64, 128, 256, 511, 1021, 2040, 4076, 8144] ...
n=10,  decanacci -> [1, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1023, 2045, 4088, 8172] ...

Three answers and Zen

http://people.canonical.com/~therve/Twisted/12.1.0pre1/

• The revival of the kqueue reactor for BSD platforms.
• epoll is now the default reactor under Linux after a fix to handle files on stdin/stdout.
• New reactors supporting GTK3 and GObject-Introspection.
• Several enhancements regarding file descriptors passing: systemd support for servers, ability to send and receive file descriptors across Unix sockets, and an AMP argument using that feature.
• Support for IPv6 literals in connectTCP.
• Persistent connections support for the new HTTP client.

def tree():
    return defaultdict(tree)

lupin = tree()
lupin["express"][3] = "stand and deliver"

It’s interesting to think about what’s going on in the above code and why it works. I really like defaultdict.

What I more often want, though, is not infinitely deep nested dictionaries like the above, but a (known) finite number of nested defaultdicts with a specific type at the final level. Here’s a tiny function I wrote to get just that:

def autovivify(levels=1, final=dict):
    return (defaultdict(final) if levels < 2 else
            defaultdict(lambda: autovivify(levels – 1, final)))

So let’s say you were counting the number of occurrences of words said by people by year, month, and day in a chat room. You’d write:

words["sam"][2012][5][25]["hello"] += 1
words["sue"][2012][5][24]["today"] += 1

Etc. It’s pretty trivial, but it was fun to think about how to do it with a function and to play with some alternatives. You could do it manually with nested lambdas:

But that gets messy quickly and isn’t nearly as much fun.

We’ll start out with a really simple example code snippet. The following contains two buttons, one which copies any text that’s added to the text control and which you can then paste elsewhere, such as in the text box, in a search engine or whatever. The other button also copies to clipboard and then closes the application after flushing the data. This is supposed to make the data available in the system clipboard even after the application is closed. Both work great on Windows, but wxGTK (i.e. the Linux version) doesn’t work in the latter case. See the bug ticket for more information.

Anyway, let’s look at the code!

import wx
 
########################################################################
class ClipboardPanel(wx.Panel):
    """"""
 
    #----------------------------------------------------------------------
    def __init__(self, parent):
        """Constructor"""
        wx.Panel.__init__(self, parent)
 
        lbl = wx.StaticText(self, label="Enter text to copy to clipboard:")
        self.text = wx.TextCtrl(self, style=wx.TE_MULTILINE)
        copyBtn = wx.Button(self, label="Copy")
        copyBtn.Bind(wx.EVT_BUTTON, self.onCopy)
        copyFlushBtn = wx.Button(self, label="Copy and Flush")
        copyFlushBtn.Bind(wx.EVT_BUTTON, self.onCopyAndFlush)
 
        sizer = wx.BoxSizer(wx.VERTICAL)
        sizer.Add(lbl, 0, wx.ALL, 5)
        sizer.Add(self.text, 1, wx.EXPAND)
        sizer.Add(copyBtn, 0, wx.ALL|wx.CENTER, 5)
        sizer.Add(copyFlushBtn, 0, wx.ALL|wx.CENTER, 5)
        self.SetSizer(sizer)
 
    #----------------------------------------------------------------------
    def onCopy(self, event):
        """"""
        self.dataObj = wx.TextDataObject()
        self.dataObj.SetText(self.text.GetValue())
        if wx.TheClipboard.Open():
            wx.TheClipboard.SetData(self.dataObj)
            wx.TheClipboard.Close()
        else:
            wx.MessageBox("Unable to open the clipboard", "Error")
 
    #----------------------------------------------------------------------
    def onCopyAndFlush(self, event):
        """"""
        self.dataObj = wx.TextDataObject()
        self.dataObj.SetText(self.text.GetValue())
        if wx.TheClipboard.Open():
            wx.TheClipboard.SetData(self.dataObj)
            wx.TheClipboard.Flush()
        else:
            wx.MessageBox("Unable to open the clipboard", "Error")
 
        self.GetParent().Close()
 
########################################################################
class ClipboardFrame(wx.Frame):
    """"""
 
    #----------------------------------------------------------------------
    def __init__(self):
        """Constructor"""
        wx.Frame.__init__(self, None, title="Clipboard Tutorial")
        panel = ClipboardPanel(self)
        self.Show()
 
 
if __name__ == "__main__":
    app = wx.App(False)
    frame = ClipboardFrame()
    app.MainLoop()

As you might have guessed, the guts of this script are in the button event handlers. The main bit is wx.TextDataObject which will store the data from the text control. Next we attempt to open the clipboard. If we’re successful, we add our text to the clipboard and then close it. The data is now there for the pasting. The second event handler does the same thing, but it Flushes to the clipboard rather than just closing it. If you wanted to copy a bitmap instead, then you’d want to use a wx.BitmapDataObject and pass it an wx.Bitmap object. Otherwise, the rest is the same.

Wrapping Up

Now you know how to use the clipboard from wxPython! Get out there and start creating something to amaze your friends and strangers alike!

Additional Resources

• The wx.Clipboard documentation
• The wxPython wiki page on the Clipboard

Python integration

We’ve integrated Telemetry into the game engine for performance evaluation purposes. As part of this, I also integrated it to the Python library. I wrote a special “TelemetryProfiler” class that we can use with sys.setprofile() so that .py excecution can be monitored. In addition, I added some select monitoring hooks into the core engine, and around such places as locks and for memory allocations.

To work well with threads and tasklets, we maded some changes to allow for global profiling. Using a flag called stackless.globaltrace, sys.setprofile() (and sys.settrace()) have global effects on both all tasklets and all threads. The profiling flag is stored in the interpreter state so that new threads will inherit it, and currently executing threads will have it set by traversing the list of existing threads. This allows us to turn profiling on and off in an executing program and monitor all that the python engine is doing.

For profiling stackless tasklets, we had to resort to trickery. Because Telemetry knows about threads, but not microthreads, we had to add code to recognize the currently running tasklet, and unwind and replay the context stack as necessary whenever at tasklet switch occurred.

Here is a snapshot of parts of a Python execution as it is displayed in the Telemetry vizualizer:

Python running on the main thread

Notice the hierarchical call graph. Tooltips will show more information about each field, such as Python source file and line number. The Mem Events bar at the bottom shows all memory allocations made and the lifetime of each of them.
Notice also the red blips. These are the GIL being ticked (there is definitely room for a faster GIL tick function. We already treat the GIL differently from other locks and an optimized Tick is simple to add. Later.)

Condition Variables.

As previously mentioned on this blog, we have to perform HTTP requests on worker threads and then communicate the results back to the main tasklet. The worker threads live in a threadpool and receive work from any client thread. Dispatching of work requests is done using the tried and tested Condition Variable model, using the threading.Condition class and an associated threading.RLock object. Workers wait on condition variables until they are woken up and see that there is work to be done, at which point they pull work off a queue, release the lock, perform the work, and then do it again.

To lower latency, it is important that this dispatching of work to the worker thread happens as soon as possible. Ideally, it should happen immediately, the worker grabbing the GIL, kicking the HTTP Request into action, releaseing the GIL and blocking. To see if things were progressing in a timely manner, I examined the process using Telemetry.

First off, I should explain what locking model we are using. The PS3 is essentially posix-like, so locks (and the GIL) are implemented using the thread_pthread.h file. The default choice in there is to use the posix semaphore API to implement the non-recursive locks Python uses. But I was quickly slapped by the engine profilers when they started seeing all those kernel transitions coming out of Python. Aqcuiring a free lock, and ticking the GIL, when these are based on a semaphore, incurs a kernel transition and scheduling and whatnot, a high price to pay when there is no contention. This lights up red in all performance measurement tools. So, we opted for using the mutex/condition implementation also provided. Why this is not the default in Python is beyond me, since it is much faster.

So, anyway, this is what waking up on a condition variable looks like in Telemetry. The worker thread has been waiting on a threading.Condition object. The main posts a request and the worker wakes up.

Waking up from the condition variable

What you see on the left is an acquire() in progress. This is the Lock object that the condition variable uses to sleep on. Then, we _acuire_restore(), then some logging code and finally the httpRequest being sent.  (Ignore the logging code and its associated GIL delay, it is only used during debugging.)
Zooming in a bit, brings more detail:

detail of wakeup

What you see here, is the GIL being acquired two times: Once to complete the Lock.acquire() call which is done in Condition.wait(), and once to re-acquire the lock associated with the condition variable (typically an RLock): To complete the condition variable protocol and return from wait() with the lock held.

Now, in a multithreaded program, the GIL is always in contention. Releasing and reaquiring the GIL two times instead of once will therefore add unnecessary latency into this code. If this second GIL aquisition were skipped, this wakeup delay would have been shortened by 10 milliseconds, kicking off the http request that much faster.

Optimization one: _acquire_cv()

The first obvious thing to do is to roll this double acquisition of a lock into one. There are two locks that need to be acquired here, the wait lock and the lock associated with the Condition object. There is no reason they can’t be both acquired in one swell foop.

I did this by adding a new method to the Lock class in thread.c: _acquire_cv() takes an additional Lock object which is always acquired before returning from the function, except when the a try-aquire fails (to support the emulated timeout behaviour.)

Now, if the outer lock is acquired on return, there is no need for an _acquire_restore() call in threading.py, so this has been replaced with a _restore() call, which merely sets the internal variables on an already acquired lock.

This is what the new Condition.wait call looks like:

    def wait2(self, timeout=None):
        if not self._is_owned():
            raise RuntimeError("cannot wait on un-acquired lock")
        waiter = _allocate_lock()
        waiter.acquire()
        self.__waiters.append(waiter)
        saved_state = self._release_save()
        gotit = 0
        try:    # restore state no matter what (e.g., KeyboardInterrupt)
            if timeout is None:
                gotit = waiter._acquire_cv(1, saved_state[0])
                if __debug__:
                    self._note("%s.wait(): got it", self)
            else:
                pass # removed for brevity
        finally:
            if gotit:
                self._restore(saved_state)
            else:
                self._acquire_restore(saved_state)

    wait = wait2

With this in place, things look a little different:

On the bottom left, two Lock regions are visible (this is when acquiring a thread.Lock object.). Once the first has been acuired, i.e. the wakeup signal recieved, it proceeds to aquire the outer lock. This succeeds once the first thread (above in the picture) has released it, in the __exit__ method of a context manager. Once that is done, a single GIL reaquisition is performed. Now, this one is fairly short and some can be longer than this, but it is clear that doing this once, rather than twice is beneficial since it reduces the thrashing of threads involved.

This simple change is really easy to implement. the _acquire_cv() function is simple, and the wait2() function is simpler than the wait() function in threading.py. This is something I’d immediately recommend for cPython.

Native Condition objects

So, is there anything else that can be done? Yes there is. The Condition class is implemented in threading.py by using Python’s native non-recursive locks (semantically, they are semaphores with a max value of 1) and juggling a queue and waking things up. This is what calling condition.wait() looks like:

calling condition.wait()

On the left, there is a lot of action where a Lock object that is used for waiting is being created and put into a list. Then there is a long stall when the initial lock is aquired. Why is that? It’s because every Lock.acquire() releases the GIL, even if the lock is uncontested. It is the nature of the internal architecture of cPython that Lock objects are GIL unaware, so the GIL manipulation must be done outside of them. So, to aquire our our new wait object, the GIL is released, the lock acquired, and then the GIL must be reaquired.

It would be possible here to use the new _acquire_cv() to do this without releasing the GIL, perhaps by passing None as its second argument. But since I’m already using Locks based on mutexes and condition variables, why not simply implement a native Condition object? Doing that would rid us of most of the annoying code in threading.Condition. No need to create a Lock object for every wait, manage a queue, and potentially leave Lock object in an aquired state when destroying them (yes, the timeout version of Condition variables in Python 3 has that race problem.) Also, we would get the built-in timeout mechanism of all condition variable implementations I’ve seen for free.

Since the native Python lock objects are non-recursive, unlike classical mutex objects as used with condition variables, blocking is implemented with an internal Condition variable for each lock. So, a native thread.Condition class’s wait() method cannot directly just pass in a mutex. Instead, we must roll some of our lock release() and acquire() release machinery into the wait() and notify() functions.

This is what the thread.Condition.wait() function looks like:

int
PyThread_cond_wait(PyThread_type_condition *cond, PyThread_type_lock lock, double timeout)
{

    /* Release the Lock, keeping the Mutex held */
     * no need to release GIL here, this is a short lived thing
     */
    pthread_lock *thelock = (pthread_lock *)lock;
    int status;
    PyTmEnter(PYTM_ENTER_IDLE, "COND");
    status = pthread_mutex_lock( &thelock->mut );
    CHECK_STATUS_COND;
    if (!thelock->locked) {
        pthread_mutex_unlock( &thelock->mut );
        PyErr_SetString(PyExc_RuntimeError, "cannot wait on un-acquired lock");
        return -1;
    }
    thelock->locked = 0;
    status  = pthread_cond_signal(&thelock->lock_released);
    if (status) {
        thelock->locked = 1;
        pthread_mutex_unlock( &thelock->mut );
        CHECK_STATUS_COND;
    }
    /* now, wait for the condition variable */
    Py_BEGIN_ALLOW_THREADS
    if (timeout >= 0) {
        struct timespec ts;
#if SN_TARGET_PS3
        sys_time_sec_t sec;
        sys_time_nsec_t nsec;
        sys_time_get_current_time(&sec, &nsec);
        ts.tv_sec = sec;
        ts.tv_nsec = nsec;
#endif /* todo: implement posix */

        ts.tv_sec += (int)timeout;
        ts.tv_nsec += (int) 1e9 * (timeout - (double)(int)timeout);
        status = pthread_cond_timedwait(cond, &thelock->mut, &ts);
        if (status == ETIMEDOUT)
            status = 0; /* don't specifically signal this */
    } else {
        status = pthread_cond_wait(cond, &thelock->mut);
    }
    /* now, reacquire the lock.  We already hold the mutex. */
    while (thelock->locked) {
        int status2 = pthread_cond_wait(&thelock->lock_released, &thelock->mut);
        if (status2) {
            if (!status)
                status = status2; /* otherwise, ignore this error */
            break;
        }
    }
    thelock->locked = 1;
    pthread_mutex_unlock( &thelock->mut );
    Py_END_ALLOW_THREADS
    PyTmLeave();
    CHECK_STATUS_COND;
    return 0;
}

Similarly, the notify() method must hold the mutex when called:

static int
_PyThread_cond_notify_some(PyThread_type_condition *cond, PyThread_type_lock lock, int all)
{
    pthread_lock *thelock = (pthread_lock *)lock;
    int status = pthread_mutex_lock( &thelock->mut );
    CHECK_STATUS_COND;
    if (!thelock->locked) {
        pthread_mutex_unlock( &thelock->mut );
        PyErr_SetString(PyExc_RuntimeError, "cannot notify on un-acquired lock");
    }
    if (all)
        status = pthread_cond_broadcast(cond);
    else
        status = pthread_cond_signal(cond);
    pthread_mutex_unlock( &thelock->mut );
    CHECK_STATUS_COND;
    return 0;
}

So, what effect does this have? This is what thread.Condition.wait() looks like in telemetry:

thread.Condition.wait()

There are some blips on the left when calling the initial predicate in the wait_for_condition() function and the call of the threading.RLock._save() method (to store the recursion state of the RLock.) There is no unnecessary flapping of the GIL.

When waking up, it looks like this when the notify() call is made from the main thread:

calling tread.Condition.wait()

And here the worker finally aquires the GIL:

GIL aquired and wait() returns

Internally, the notify() call causes the native condition variable to wake up. The delay until it tries to reaquire the GIL is the when it is trying to aquire the associated RLock object. Then once the GIL is finally aqcuired, there is a call to _restore (to restore the recursion state of the RLock) and a finall call to the predicate function to make sure that the the external conditions are now satisfied. In the whole wait() and notify() process, there is no GIL trashing whatsoever.

Conclusion

It is fairly straightforward to modify the native locks in Python to cater to the special semantics required for threading.Condition objects, by adding a special method to acquire two locks at once, and also by acquiring a lock that is known to be uncontested by not releasing the GIL.

For those lock implementations that rely on mutexes and condition variables, it is also simple to provide a native implementation of a Condition object.  Currently this applies to pthreads only, but I have a condition variable / mutex locking implementation waiting in the wings for Windows, which would make the same thing possible there (and in fact our CCP branch of cPython 2.7 does just that).

threading.py can easily use these new features if available and fall back to the old behaviour if not.

The only semantic change when providing a native Condition variable is that then the python condition variable may become subject to spurious wakeup. While python has never promised that spurious wakeup cannot occur, the implementation in threading.py has been free of it.  However, the condition variable, as derived from the monitor concept, has always been careful to allow for spurious wakeups as a possibility.  There are good reasons for it, and since stolen wakeups are always a possibility regardless, and the correct way of using wait() is in a loop, checking an external condition, it does not matter to the conforming programmer.

We’ll be going forward by using the native Condition implementation on the PS3 to increase synergy, and drill down to better leverage towards a procured paradigm shift vis-à-vis a deliverable home run.

Why a filesystem?

If you've been doing MongoDB for a while, you may have heard about the 16 MB document size limit. When I started using MongoDB (around version 0.8), the limit was actually 4 MB. What this means is that everything is working just fine, your system is screaming fast, until you try to create a document that's 4.001 MB and boom nothing works any more. For us at SourceForge, what that meant was that we had to restructure our schema and use less embedding.

But what if it's not something that can be restructured? Maybe your site allows users to upload large attachments of unknown size. In such cases you probably can get away with using a Binary field type and crossing your fingers, but a better solution, in my opinion, is to actually store the contents your upload in a series of documents (let's call them "chunks") of limited size. Then you can tie them all together with another document that specifies all the file metadata.

GridFS to the rescue

Well, that's exactly what GridFS does, but it does it with a nice API with a few more bells and whistles than you'd probably build on your own. It's important to note that GridFS, implemented in all the MongoDB language drivers, is a convention and an api, not something that's provided natively by the server. As far as the server is concerned, it's all just collections and documents.

The GridFS schema

GridFS actually stores your files in two collections, named by default fs.files and fs.chunks, although you can change the fs to something else if you'd like. The fs.files collection is where reading or writing a file begins. A typical fs.files document looks like the following (credit):

{
  // unique ID for this file
  "_id" : <unspecified>,
  // size of the file in bytes
  "length" : data_number,
  // size of each of the chunks.  Default is 256k
  "chunkSize" : data_number,
  // date when object first stored
  "uploadDate" : data_date,
  // result of running the "filemd5" command on this file's chunks
  "md5" : data_string
}

The fs.chunks collection contains all the data for your files:

{
  // object id of the chunk in the _chunks collection
  "_id" : <unspecified>,
  // _id of the corresponding files collection entry
  "files_id" : <unspecified>,
  // chunks are numbered in order, starting with 0
  "n" : chunk_number,
  // the chunk's payload as a BSON binary type
  "data" : data_binary,
}

In the Python gridfs package (included with the pymongo driver), several other fields are inserted as well:

filename

This is the 'human' name for the file, which may be path-delimited to simulate directories.

contentType

This is the mime-type of the file

encoding

This is the unicode encoding used for text files.

You can also add in your own attributes to files. At SourceForge, we used things like project_id or forum_id to allow the same filename to be uploaded to multiple places on the site without worrying about namespace collisions. To keep your code future-proof, you should put any custom attributes inside an embedded metadata document, just in case the gridfs spec expands to incorporate more fields.

Using GridFS

So with all that out of the way, how to you actually use GridFS? It's actually pretty straightforward. The first thing you need is a reference to a GridFS filesystem:

>>> import pymongo
>>> import gridfs
>>> conn = pymongo.Connection()
>>> db = conn.gridfs_test
>>> fs = gridfs.GridFS(db)

Basic reading and writing

Once you have the filesystem, you can start putting stuff in it:

>>> with fs.new_file() as fp:
...     fp.write('This is my new file. It is teh awezum!')

Let's examine the underlying collections to see what actually happened:

>>> list(db.fs.files.find())
[{u'length': 38,
  u'_id': ObjectId('4fbfa7b9fb72f096bd000000'),
  u'uploadDate': datetime.datetime(2012, 5, 25, 15, 39, 37, 55000),
  u'md5': u'332de5ca08b73218a8777da69293576a',
  u'chunkSize': 262144}]
>>> list(db.fs.chunks.find())
[{u'files_id': ObjectId('4fbfa7b9fb72f096bd000000'),
  u'_id': ObjectId('4fbfa7b9fb72f096bd000001'),
  u'data': Binary('This is my new file. It is teh awezum!', 0),
  u'n': 0}]

You can see that there's really nothing surprising or mysterious happening there; it's just mapping the filesystem metaphor onto MongoDB documents. In this case, our file was small enough that it didn't need to be split into chunks. We can force split it by specifying a small chunkSize when creating the file:

>>> with fs.new_file(chunkSize=10) as fp:
...     fp.write('This is file number 2. It should be split into several chunks')
...
>>> fp
<gridfs.grid_file.GridIn object at 0x1010f5950>
>>> fp._id
ObjectId('4fbfa8ddfb72f0971c000000')
>>> list(db.fs.chunks.find(dict(files_id=fp._id)))
[{... u'data': Binary('This is fi', 0), u'n': 0},
 {... u'data': Binary('le number ', 0), u'n': 1},
 {... u'data': Binary('2. It shou', 0), u'n': 2},
 {... u'data': Binary('ld be spli', 0), u'n': 3},
 {... u'data': Binary('t into sev', 0), u'n': 4},
 {... u'data': Binary('eral chunk', 0), u'n': 5},
 {... u'data': Binary('s', 0), u'n': 6}]

Now, if we actually want to read the file as a file, we'll need to use the gridfs api:

>>> with fs.get(fp._id) as fp_read:
...     print fp_read.read()
...
This is file number 2. It should be split into several chunks

Treating it more like a filesystem

There are several other convenience methods bundled into the GridFS object to give more filesystem-like behavior. For instance, new_file() takes any number of keyword arguments that will get added onto the fs.files document being created:

>>> with fs.new_file(
...     filename='file.txt', 
...     content_type='text/plain', 
...     my_other_attribute=42) as fp:
...     fp.write('New file')
...
>>> fp
<gridfs.grid_file.GridIn object at 0x1010f59d0>
>>> db.fs.files.find_one(dict(_id=fp._id))
{u'contentType': u'text/plain',
 u'chunkSize': 262144,
 u'my_other_attribute': 42,
 u'filename': u'file.txt',
 u'length': 8,
 u'uploadDate': datetime.datetime(2012, 5, 25, 15, 53, 1, 973000),
 u'_id': ObjectId('4fbfaaddfb72f0971c000008'), u'md5':
 u'681e10aecbafd7dd385fa51798ca0fd6'}

Better would be to encapsulate my_other_attribute into the metadata property:

>>> with fs.new_file(
...     filename='file2.txt', 
...     content_type='text/plain', 
...     metadata=dict(my_other_attribute=42)) as fp:
...     fp.write('New file 2')
...
>>> db.fs.files.find_one(dict(_id=fp._id))
{u'contentType': u'text/plain',
 u'chunkSize': 262144,
 u'metadata': {u'my_other_attribute': 42},
 u'filename': u'file2.txt',
 u'length': 10,
 u'uploadDate': datetime.datetime(2012, 5, 25, 15, 54, 5, 67000),
 u'_id':ObjectId('4fbfab1dfb72f0971c00000a'),
 u'md5': u'9e4eea3dec28d8346b52f18086437ac7'}

We can also "overwrite" files by filename, but since GridFS actually indexes files by _id, it doesn't get rid of the old file, it just versions it:

>>> with fs.new_file(filename='file.txt', content_type='text/plain') as fp:
...     fp.write('Overwrite the so-called "New file"')
...

Now, if we want to retrieve the file by filename, we can use get_version or get_last_version:

>>> fs.get_last_version('file.txt').read()
'Overwrite the so-called "New file"'
>>> fs.get_version('file.txt', 0).read()
'New file'

Since we've been uploading files with a filename property, we can also list the files in gridfs:

>>> fs.list()
[u'file.txt', u'file2.txt']

We can also remove files, of course:

>>> fp = fs.get_last_version('file.txt')
>>> fs.delete(fp._id)
>>> fs.list()
[u'file.txt', u'file2.txt']
>>> fs.get_last_version('file.txt').read()
'New file'

Note that since only one version of "file.txt" was removed, we still have a file named "file.txt" in the filesystem.

Finally, gridfs also provides convenience methods for determining if a file exists and for quickly writing a short file into grifs:

>>> fs.exists(fp._id)
False
>>> fs.exists(filename='file.txt')
True
>>> fs.exists({'filename': 'file.txt'}) # equivalent to above
True
>>> fs.put('The quick brown fox', filename='typingtest.txt')
ObjectId('4fbfad74fb72f0971c00000e')
>>> fs.get_last_version('typingtest.txt').read()
'The quick brown fox'

So that's the whirlwind tour of GridFS. I'd love to hear how you're using GridFS in your project, or if you think it might be a good fit, so please drop me a line in the comments.

Subscribe to our newsletter

• Logsna – a sane log output format (Ruslan Spivak’s blog)
• Python.org Redesign Request For Proposals (Jesse Noller)
• What Generators are for (Simeon Franklin)
• Recent Windows Changes in Python 3.3
• I think it’s exciting that there’s now an RFP out for a redesign of the Python website
• Nick Coghlan had a pretty interesting article about all the resources there are now for deployment, docs and testing

I don’t want to overload you on links and I know this one is older, but I have been really interesting in following Katie C’s gaming adventures. So I’ll leave you on that note. I hope you enjoy the articles as much as I did.

Quick Review

• Why I picked it up: Partly because Packt offered me a review copy and partially because I wanted to see how web2py differed from django and TurboGears
• Why I finished it: I finished it out of duty to the publisher and my audience. Cookbooks are hard to read straight through and tend to be kind of boring.
• I’d give it to: Beginner to intermediate developers wanting to learn some new tricks with web2py

Book Formats

Paperback, Kindle (mobi) or PDF and epub (from Packt)

Book Contents

Chapter 1 is about deploying web2py to various backends. While a little dull, I think this section could be very helpful for deployment. The second chapter is mostly about improving on the “scaffolds” application that comes with web2py. Chapter 3 covers how web2py interacts with databases and various CSV manipulations. Number four covers advanced forms such as multiple forms on one page. I don’t find javascript very interesting right now, so chapter 5′s topic of AJAX didn’t do much for me. Chapter 6 covers some third party libraries, like Twitter connectivity, customizing logging, etc. In 7 we learn about web services and in 8 we go over authentication and authorization techniques. Chapter 9 is all about custom URL routing. For 10, we learn various ways to create PDF reports. The last chapter is a collection of recipes that didn’t really fit anywhere else and are pretty random. From PDB integration to using web2py with wxPython, it covers a wide array of topics.

Review

We’ll start off with some of the bad as I like to finish with the good. I found lots of silly issues that a normal spell check program should have flagged. The writing isn’t compelling either, but most cookbooks I’ve read are that way. It may be dry, but it does a decent job of explaining most of the recipes. Some are explained more than others. This is also pretty standard in a cookbook. I’m not knocking the authors for the poor editing as that’s not their job and I can tell that several if not all are not native English speakers. I’m sure they did their best.

What I liked most is that this book actually has recipes that I not only thought were interesting but that I could see myself actually using. There are recipes for integrating PayPal payments, creating CAPTCHAs, building Facebook and Reddit clones, various web service consumer recipes, and some debugging stuff at the end. I was especially interested in the PayPal stuff as I just haven’t seen much in the Python web framework world about payment systems in general.

The PDF integration was also interesting in that web2py can use reportlab, LaTeX or pyfpdf. I’d never heard of the last one, so that was interesting in and of itself. There really isn’t the space to go over all the recipes that caught my eye. I will point out that most of the examples are incomplete, probably because there just isn’t space. Fortunately you can download the full source code from Packt’s website. I should also note that some chapters seem to have more javascript than Python. That may be expected, but I was still a little surprised by it. Because the code is incomplete, I didn’t try to run it from the book. I don’t really recommend that anyway when it’s not all there.

Developers who would get the most out of this book are probably beginners to intermediate web2py programmers who would like to increase their skill set. I think there are lots of nifty tricks to be learned from this book and I hope to try them out at some point.

I've released Powerhose 0.4 at PyPI - http://pypi.python.org/pypi/powerhose/0.4, and this new version has a few changes that are worth speaking of.

import traceback, sys, gc

def dump_stacks():
    dump = []

    # threads
    threads = dict([(th.ident, th.name)
                        for th in threading.enumerate()])

    for thread, frame in sys._current_frames().items():
        dump.append('Thread 0x%x (%s)\n' % (thread, threads[thread]))
        dump.append(''.join(traceback.format_stack(frame)))
        dump.append('\n')

    # greenlets
    try:
        from greenlet import greenlet
    except ImportError:
        return dump

    # if greenlet is present, let's dump each greenlet stack
    for ob in gc.get_objects():
        if not isinstance(ob, greenlet):
            continue
        if not ob:
            continue   # not running anymore or not started
        dump.append('Greenlet\n')
        dump.append(''.join(traceback.format_stack(ob.gr_frame)))
        dump.append('\n')

    return dump

$ powerhose-broker --check
.. checks if the broker is alive and running, prints its pids...

$ powerhose-broker --purge-ghosts
.. kill any "ghost" broker...

Software development is traditionally a field dominated by men. The presence of women in programming is commanding more attention at conferences and events, particularly in recent years. There are many initiatives within the Python community to promote involvement, such as the group Women Who Code, and the long-term benefits of efforts like these cannot be overstated. At Caktus, three of the ten current members of our Django development staff are female, and we are actively working to further equalize the gender balance through future hires.

One part of closing the gender gap in programming communities involves creating a safe and inclusive environment for all parties, both in the workplace and at community events. This begins with strong employment policies on harassment — which we have implemented here at Caktus — and also extends to implementing policies at conferences or other events relevant to the development community. This blog post comes out of several recent events in the broader tech community that Caktus is a part of. This includes two blog posts that drew a lot of attention, and two corporate events that demonstrated clearly for us how pervasive sexism still is in our industry.

In March, Katie Cunningham, a fellow Django developer and friend made through DjangoCon, wrote an excellent and widely read blog post titled "Lighten Up." The post details a story that is all too common in our industry, that of ongoing, but usually subtle sexist jokes and comments that serve only, whether intentionally or not, to gradually demoralize and discourage those to whom they are addressed. She discusses how the typical response, should someone get upset by such a "joke" or comment, is simply to "lighten up" and "stop being so sensitive." I can see how that would get really old. Fast. In fact, just in reading up for writing this post, I was blown away by the number of comments in response to posts such as Katie's that fit exactly that description.

Just a couple weeks ago, the computer manufacturer Dell hosted a summit in Copenhagen. Dell hired a "comedian" to MC the event, who proceeded to entertain the attendees with a variety of sexist jokes, acclaiming the "success" of the IT community as indicated by its male-dominated culture, and included a suggestion to the mostly male attendees that they go home and tell their partners to "shut up bitch." Dell has finally put out an apology about what happened, but the consensus seems to be that it's weak at best and, given its tardiness, lost any sincerity it may once have had.

In closing, I’d like to draw attention to one other aspect of the software development community, particularly at conferences, that often tends to aggravate any pre-existing sexism that might exist in the community. Last month, Ryan Funduk wrote an extensive and much-debated blog post, titled "Our Culture of Exclusion," about the pervasiveness of drinking at many programming conferences. He also discusses how, all too often, these environments are ripe with sexist or racist jokes (or worse) that quickly erode any efforts to further diversify such communities. One example that he highlights is the daily deal API provider Sqoot, who alienated and offended many when they suggested that women are better fit to serve beer than to program. Sqoot made several apologies about this unfortunate advertisement, but many of them still fell short and only worsened the offense, as this response from Gayle McDowell aptly sums up:

Your language, as far as I understand it, is making the assumption that all coders are male, that all are straight, that women are sexual objects offered as a reward (HIGHLY inappropriate in a professional context), and that the women are there to serve them.

She goes on to suggest an appropriate response if and when events like this do happen:

People screw up sometimes. It happens. A TON of people (both men and women) have done equally offensive things. // But when you do it, you need to own up to it and think about what you've done. Not lie about it, as you're doing now.

Clearly, we need to have policies in place for handling instances of sexist or racist jokes and sexual harassment. When they do happen, McDowell says, one needs to take responsibility and own up to the mistakes one has made in an honest way.

The bar has been set, and I think it’s worth noting that several lessons can be taken from these events and applied to our own communities. Caktus is a long-time sponsor of several conferences relevant to the work that we do. Along with this blog post, Caktus is asking conference organizers and other sponsors to join us in the following effort: Moving forward, Caktus will require that a zero-tolerance sexual harassment policy is established and enforced by the organizers of any conference that we sponsor or attend. We want to ensure that our community events are safe, welcoming, and supportive for all of our colleagues — both male and female. While drinking can certainly make things worse, it is not the root of the problem. To overcome these issues, we need to talk about them more, continue to raise awareness, and treat all people in our workplaces and communities with the professionalism and respect that they deserve.

I’m pretty excited! Now if only the Python Imaging Library would convert too…

http://wxpython.org/Phoenix/ItsAlive/

PyFrameObject *
PyFrame_New(PyThreadState *tstate, PyCodeObject *code, PyObject *globals,
PyObject *locals)
...
if (code->co_zombieframe != NULL) {
f = code->co_zombieframe;
code->co_zombieframe = NULL;
_Py_NewReference((PyObject *)f);
assert(f->f_code == code);
}

Intrigued by the name, I examined the header where it is defined, code.h:

...
void *co_zombieframe; /* for optimization only (see frameobject.c) */
...
} PyCodeObject;

It turns out that for every PyCodeObject object that has been executed, a PyFrameObject of a suitable size is cached and kept with the code object. Now, caching is fine and good, but this cache is unbounded. Every code object has the potential to hang on to a frame, which may then never be released.
Further, there is a separate freelist cache for PyFrameObjects already, in case a frame is not found on the code object:

if (free_list == NULL) {
f = PyObject_GC_NewVar(PyFrameObject, &PyFrame_Type,
extras);
if (f == NULL) {
Py_DECREF(builtins);
return NULL;
}
}
else {
assert(numfree > 0);
--numfree;
f = free_list;
free_list = free_list->f_back;
...

Always concious about memory these days, I tried disabling this in version 3.3 and running the pybench test. I was not able to see any conclusive difference in execution speed.

Update:

Disabling the zombieframe on the PS3 shaved off some 50k on startup.  Not the jackpot, but still, small things add up.

——————————————————————————-
PYBENCH 2.1
——————————————————————————-
* using CPython 3.3.0a3+ (default, May 23 2012, 20:02:34) [MSC v.1600 64 bit (AMD64)]
* disabled garbage collection
* system check interval set to maximum: 2147483647
* using timer: time.perf_counter
* timer: resolution=2.9680909446810176e-07, implementation=QueryPerformanceCounter()

——————————————————————————-
Benchmark: nozombie
——————————————————————————-

Rounds: 10
Warp: 10
Timer: time.perf_counter

Machine Details:
Platform ID: Windows-7-6.1.7601-SP1
Processor: Intel64 Family 6 Model 26 Stepping 5, GenuineIntel

Python:
Implementation: CPython
Executable: D:pydevhgcpython2pcbuildamd64python.exe
Version: 3.3.0a3+
Compiler: MSC v.1600 64 bit (AMD64)
Bits: 64bit
Build: May 23 2012 20:02:34 (#default)
Unicode: UCS4

——————————————————————————-
Comparing with: zombie
——————————————————————————-

Rounds: 10
Warp: 10
Timer: time.perf_counter

Machine Details:
Platform ID: Windows-7-6.1.7601-SP1
Processor: Intel64 Family 6 Model 26 Stepping 5, GenuineIntel

Python:
Implementation: CPython
Executable: D:pydevhgcpython2pcbuildamd64python.exe
Version: 3.3.0a3+
Compiler: MSC v.1600 64 bit (AMD64)
Bits: 64bit
Build: May 23 2012 20:00:42 (#default)
Unicode: UCS4

Test minimum run-time average run-time
this other diff this other diff
——————————————————————————-
BuiltinFunctionCalls: 51ms 52ms -3.3% 52ms 53ms -2.0%
BuiltinMethodLookup: 33ms 33ms +0.0% 34ms 34ms +0.8%
CompareFloats: 50ms 50ms +0.1% 50ms 50ms +0.4%
CompareFloatsIntegers: 99ms 98ms +0.8% 99ms 99ms +0.6%
CompareIntegers: 77ms 77ms -0.5% 77ms 77ms -0.3%
CompareInternedStrings: 60ms 60ms +0.0% 61ms 61ms -0.1%
CompareLongs: 46ms 45ms +1.5% 46ms 45ms +1.2%
CompareStrings: 61ms 59ms +3.6% 61ms 59ms +3.6%
ComplexPythonFunctionCalls: 60ms 58ms +3.3% 60ms 58ms +3.2%
ConcatStrings: 48ms 47ms +2.4% 48ms 47ms +2.1%
CreateInstances: 58ms 57ms +1.3% 59ms 58ms +1.3%
CreateNewInstances: 43ms 43ms +1.1% 44ms 44ms +1.1%
CreateStringsWithConcat: 79ms 79ms -0.3% 79ms 79ms -0.1%
DictCreation: 71ms 71ms +0.4% 72ms 72ms +1.0%
DictWithFloatKeys: 72ms 70ms +2.1% 72ms 71ms +1.8%
DictWithIntegerKeys: 46ms 46ms +0.7% 46ms 46ms +0.4%
DictWithStringKeys: 41ms 41ms +0.0% 41ms 41ms -0.1%
ForLoops: 35ms 37ms -4.0% 35ms 37ms -4.0%
IfThenElse: 64ms 64ms -0.1% 64ms 64ms -0.4%
ListSlicing: 49ms 50ms -1.0% 53ms 53ms -0.8%
NestedForLoops: 54ms 51ms +6.7% 55ms 51ms +6.7%
NestedListComprehensions: 54ms 54ms -0.7% 54ms 55ms -2.2%
NormalClassAttribute: 94ms 94ms +0.1% 94ms 94ms +0.1%
NormalInstanceAttribute: 54ms 54ms +0.3% 54ms 54ms +0.2%
PythonFunctionCalls: 58ms 57ms +0.8% 58ms 58ms +0.6%
PythonMethodCalls: 65ms 61ms +6.3% 66ms 62ms +5.9%
Recursion: 84ms 85ms -1.0% 85ms 85ms -0.9%
SecondImport: 74ms 76ms -2.5% 74ms 77ms -3.5%
SecondPackageImport: 75ms 78ms -3.8% 76ms 79ms -3.9%
SecondSubmoduleImport: 163ms 169ms -3.4% 164ms 170ms -3.3%
SimpleComplexArithmetic: 43ms 43ms +1.0% 43ms 43ms +1.0%
SimpleDictManipulation: 80ms 78ms +2.2% 81ms 79ms +2.4%
SimpleFloatArithmetic: 42ms 42ms +0.1% 42ms 42ms -0.0%
SimpleIntFloatArithmetic: 52ms 53ms -1.2% 52ms 53ms -1.1%
SimpleIntegerArithmetic: 52ms 52ms -0.7% 52ms 53ms -0.8%
SimpleListComprehensions: 45ms 45ms -0.2% 45ms 45ms +0.3%
SimpleListManipulation: 44ms 46ms -4.0% 44ms 46ms -3.9%
SimpleLongArithmetic: 32ms 32ms -0.9% 32ms 32ms -0.1%
SmallLists: 58ms 57ms +1.2% 58ms 67ms -12.8%
SmallTuples: 64ms 65ms -0.5% 65ms 65ms -0.2%
SpecialClassAttribute: 148ms 149ms -0.8% 149ms 150ms -1.0%
SpecialInstanceAttribute: 54ms 54ms +0.2% 54ms 54ms +0.0%
StringMappings: 120ms 117ms +2.5% 120ms 117ms +2.5%
StringPredicates: 62ms 62ms +0.9% 62ms 62ms +1.0%
StringSlicing: 69ms 68ms +1.6% 69ms 68ms +2.1%
TryExcept: 37ms 37ms +0.0% 37ms 37ms +0.5%
TryFinally: 40ms 37ms +6.7% 40ms 37ms +6.5%
TryRaiseExcept: 19ms 20ms -1.0% 20ms 20ms -0.4%
TupleSlicing: 65ms 65ms +0.5% 66ms 65ms +1.2%
WithFinally: 57ms 56ms +1.9% 57ms 56ms +2.1%
WithRaiseExcept: 53ms 53ms +0.3% 54ms 54ms -0.8%
——————————————————————————-
Totals: 3154ms 3145ms +0.3% 3176ms 3177ms -0.0%

(this=nozombie, other=zombie)

I’m going to remove this weird, unbounded cache from the python interpreter we use on the PS3.

Do you ever get the urge to kill? How many of us cringe whenever we see these words? Lately I’ve been spending a lot of time developing pythonpackages.com, (now running on heroku!) during which time I see a lot of these kinds of packages being released:

I kid about the killing part, but seriously: this is a problem. Fortunately for us, our PyPI overloads see fit to occasionally remove these packages, and for this I am grateful. I love seeing this:

I mean it makes me dance-around-the-room happy! Ahem. But are they really all gone? Close enough. A quick crate.io search now shows only 2 packages instead of 4 pages of results:

Hallelujah! But is this the best we can do? I know that some well-meaning person wrote a book containing the example that is leading some poor, misguided souls to spam PyPI (if only the author listed the test site instead: http://testpypi.python.org/pypi). And I have to assume that this was just some terrible mistake. But do we all have to live with this mistake?

I’m asking because I honestly don’t know the answer. I remember when I started pythonpackages.com, the Deliverance documentation was being updated something like every 5 minutes (kidding again, but it was frequent enough to be annoying). After grousing about it in public, it stopped happening!

I wonder if some good natured grousing about our friends (read: enemies) the simple printers of nested lists will do the same?

Things to take away from the experience:

• Try to stay on topic! I actually found a second issue with the paragraph I was fixing in the devguide and that probably should have gone in a separate bug report.
• Number your patches! I don’t know why I didn’t think of that, but Eli told me I should do that in the future to make it less confusing for the committer. That was a face palm moment.

I’ve been reading some of the supposedly “easy bugs” and trying to figure out where else I can help. I already spotted another typo in the docs that are included with Python itself which I’ll probably try to fix. Of course, I want to actually contribute to the code, not just the documentation, but I am probably more likely to be able to find documentation bugs I can help with. Hopefully with more experience I’ll be able to contribute more effectively. Happy hacking my fellow Pythoneers!

Just because I have to use a callback-oriented style on the client doesn't mean I want to use a callback-oriented style on the server. Now, before anyone gets all upset and tells me that I don't know the difference between async and a kitchen sink, let me explain :)

The client is necessarily an event-oriented place. If I don't know which button the user is going to press, it makes a lot of sense to use a different callback for each button. The server is different. If I'm waiting for the result of a database query before I can continue processing a request, it sure is convenient to just block and wait.

My key point is that it's important to separate what style you want to code with and what performance and scalability characteristics you want. You shouldn't necessarily pick a callback-oriented style just because you want the performance and scalability characteristics of asynchronous networking APIs.

My favorite two examples are gevent and Erlang, but Go is similar. When you code using gevent or Erlang, your code looks like synchronous, blocking code. However, below the covers, they use asynchronous networking APIs. Now, before anyone tells me that it's impossible, buggy, or that it'll never work, let me point out that these tricks have been in production for decades at Ericsson, Yahoo Groups, and IronPort Cisco.

Furthermore, I should point out that asynchronous networking APIs aren't a perfect fit for every problem. For instance, if your goal is to send 10 gigabytes of information to another server, it turns out that synchronous networking APIs will actually outperform asynchronous networking APIs. The reason asynchronous networking APIs are so popular is because they can handle a larger number of clients than synchronous networking APIs can and because they use less memory than a large number of threads, which each have to have their own stack. gevent and Erlang can handle a large number of clients, don't use up much memory, and don't require a real OS-level stack per client.

So what's my problem with the callback-oriented style? I find it a lot harder to read. I've coded projects in Twisted, Node.js, etc., and I prefer the gevent approach. You get roughly the same performance and scalability characteristics, but with much easier to read code. Of course, what's readable to me may not be readable to other people. I've met people who are perfectly happy using Twisted Web 1 and don't think that callback-oriented code poses any real challenge.

If you're interested in hearing more about my thoughts on async and concurrency, check out my other blog posts, which include a link to my Dr. Dobb's Journal article on Python concurrency.

You can see the RFP here: http://pythonorg-redesign.readthedocs.org/en/latest/

It’s taken me several years of false starts, other attempts (including skunkworks attempts), political and social discussions, and the hard work of many to make this come to fruition. Now, we can only sit back and hope that we see some amazing proposals from the community and others.

I sincerely hope this will be successful, and that we will see a modern, well designed Python.org that showcases not only the language, but the vibrant, open, welcoming and active community we are all part of. 

The book I am reviewing has many examples which can easily be applied to real-world web applications, which I praise. As I have been reading through the book, I have also been playing around and exploring Web2Py in general. It has some very interesting design choices and features. It is definitely good for someone who can never remember what to import, as there is rarely any need to import anything in web2py, besides when you need to import a 3rd party Python module. At first, this design choice caught me off guard, as in Python, I am used to explicitly importing my modules for use. Web2Py includes it's technical reference with the application itself, in the form of Epydoc. I am not terribly fond of this documentation format, but it is how I learned Pyjamas, as it uses the same documentation system.

Web2Py has some interesting magic, which frankly does make web development a breeze. Generating forms and cruds is literally a one-liner, further customization can be made by using more lines of code. Web2Py also comes with a nice default template, and the application wizard, has many additional templates for use as well. This feature is definitely well welcomed in my book. Sure, it implies that tons of websites made with Web2Py will have the same look and feel, but is that such a bad thing? It just means that visitors will know where most resources can be found, and how to use the included form widgets. Think of Wordpress, most Wordpress sites I visit keep the included wordpress theme. Blogspot is another example of a set of websites which share a common theme. The included themes allow one to get to developing the website as quickly as possible without having to worry about how it will look and feel, but rather focus on the functionality. I am more of a functionality person, not a look-and-feel type of person.

All in all, I am very excited to be trying out Web2Py for a second time, and do hope that when I complete this book, I can both provide a review of the book itself, as well as the framework in general. Neither of these reviews will be compared to anything, and be strictly of the book and framework itself. I am thinking of making a future website using Web2Py to see where it goes. I will most likely keep my blog running on Django, as I have used many Django-specific features which would be otherwise troublesome to migrate over. These articles contain Django template code, for example.

This version requires pip installing the following into your virtualenv.

• pip install django-crispy-forms so we can do Python driven layouts.
• pip install django-floppyforms so we get HTML5 elements for free.
• pip install django-recaptcha to do the RECAPTCHA work.

Don't forget to add the app to your INSTALLED_APPS in settings.py:

INSTALLED_APPS += (
    'crispy_forms',
    'floppyforms',
    'captcha',
)

Generate your KEYs from the RECAPTCHA site and add them in settings.py:

RECAPTCHA_PUBLIC_KEY = '6LcVu9ESAAAAANVWwbM5-PLuLES94GQ2bIYmSNTG'
RECAPTCHA_PRIVATE_KEY = '6LcVu9ESAAAAAGxz7aEIACWRa3CVnXN3mFd-cajP'

In myapp.forms.py:

from captcha.fields import ReCaptchaField  # Only import different from yesterday
from crispy_forms.helper import FormHelper
from crispy_forms.layout import Submit
import floppyforms as forms

class ContactForm(forms.Form):

    name = forms.CharField(required=True)
    email = forms.EmailField(required=True)
    subject = forms.CharField(required=True)
    message = forms.CharField(widget=forms.Textarea)
    captcha = ReCaptchaField()  # Only field different from yesterday

    def __init__(self, *args, **kwargs):
        self.helper = FormHelper()
        self.helper.add_input(Submit('submit', 'Submit'))
        super(ContactForm, self).__init__(*args, **kwargs)

In myapp.views.py:

# Unchanged from yesterday. :-)
from django.conf import settings
from django.core.mail import send_mail
from django.views.generic import FormView

from myapp.forms import ContactForm

class ContactFormView(FormView):

    form_class = ContactForm
    template_name = "myapp/email_form.html"
    success_url = '/email-sent/'

    def form_valid(self, form):
        message = "{name} / {email} said: ".format(
            name=form.cleaned_data.get('name'),
            email=form.cleaned_data.get('email'))
        message += "\n\n{0}".format(form.cleaned_data.get('message'))
        send_mail(
            subject=form.cleaned_data.get('subject'),
            message=message,
            from_email='contact-form@myapp.com',
            recipient_list=[settings.LIST_OF_EMAIL_RECIPIENTS],
        )
        return super(ContactFormView, self).form_valid(form)

In templates/myapp/email_form.html:

{# Also unchanged from yesterday. :-)  #}
{% extends 'base.html' %}
{% load crispy_forms_tags %}

{% block title %}Send an email{% endblock %}

{% block content %}
    <div class="row">
        <div class="span6">
            <h1>Send an email</h1>
            {% crispy form form.helper %}
        </div>
    </div>
{% endblock %}

{% block extrajs %}
<script src="{{ STATIC_URL }}js/jquery-1.7.1.min.js"></script>
<script type="text/javascript">
$(function() {
    $('#id_name').focus()
});
</script>
{% endblock %}

Getting Started

First off, you need to get an account with the Bug Tracker for Python. If you hope to become a core developer, then you’ll need to make sure your username follows their guidelines, which are really simple:


firstname.lastname


Once you’ve got that, you can start looking for something to patch. There’s a link that says “Easy issues” that is a good starting place. You can also do a search for a component that you’re competent in using and see if there are any bugs in there that you think you can fix. Once you find something, you’ll need to make sure you update your local repo and then read the devguide’s patch page.

Creating the Patch

Assuming you have the necessary repository checked out on your local machine, all you need to do is go edit the appropriate file. In my case, I had to check out the devguide (which you can read about here) and edit the setup.rst file. If you’re editing Python code, then you’ll have to conform to PEP8. Once I finished editing the file, I saved my changes and then had to use Mercurial to create the patch. Here’s the command I used per the Python patch instructions.


hg diff > setup.patch


And here is the contents of that patch file:


diff -r b1c1d15271c0 setup.rst
--- a/setup.rst Tue May 22 00:33:42 2012 +0200
+++ b/setup.rst Tue May 22 13:55:09 2012 -0500
@@ -173,7 +173,7 @@
To build from the Visual Studio GUI, open pcbuild.sln to load the project
files and choose the Build Solution option from the Build menu, often
associated with the F7 key. Make sure you have chosen the "Debug" option from
-the build configuration drop-down first.
+the configuration toolbar drop-down first.

Once built you might want to set Python as a startup project. Pressing F5 in
Visual Studio, or choosing Start Debugging from the Debug menu, will launch

Now that we have a patch we need to submit it!

Submitting a Patch

Put your shields up, we’re going in! Submitting a patch is a little daunting. What will people think of you? I suspect if you plan to work on something major, then you better start growing some thick skin. In my case, I’m going to submit a really simple typo fix, so I’m hoping that sort of thing isn’t flame-worthy. Then again, this is my first patch, so I may submit it in a completely erroneous way. Since my patch will be for something (presumably) new, I did a quick search to make sure it hadn’t already been reported. Seeing nothing, I clicked the “Create New” link with some trepidation and choose the “devguide” as my component. I also chose the latest version of Python. I don’t see anything in the devguide that says it applies to just one set of Python versions, so I’m just going to leave it at that. I didn’t really see a “type” that fit a devguide edit, so I left that blank for my betters to fix. Finally, I attached my patch file to the bug ticket. You can see my bug ticket here if you like.

When contributing a patch to Python, you should fill out a contributor agreement form which allows the Python Software Foundation to license your code for use with Python while you get to keep the copyright. Yes, you too can become famous just for writing Python code! Assuming people read the source or those acknowledgement pages.

Wrapping Up

I don’t know what will happen to my rather lame contribution. Maybe it’ll get accepted, maybe not. But I think I’ll spend some time trying to figure out some other bugs and just see if there’s anything I can do to help out the Python community. Feel free to join me on this adventure!

OSX’s iTerm 2, and maybe some other terminal applications, support ANSI control sequence extensions which allow shell to set the color of the terminal tab.

Below is a Python script which

• Randomizes a color based on the server host name. The same hostname always results to the same color.
• The color is randomized in HSL color space, so that only the hue component varies and saturation and lightness are locked. This prevents the creation of ugly color combinations like black text on black tab background.

Note: The effect can be also applied on terminal windows  – for those who don’t use tabs.

The effective result is that

• You learn to identify terminal tabs by the color
• You can much more faster to switch between tabs, because you can visually pick up the terminal without needing to be able to read the text on it or remember its location in the list

Note: If your puny terminal does not support setting the color of window decorations, you can always set the terminal background color. This is useful e.g. if you want to red background for danger zone ™ when you are logged in as root on the production server 23:00 Friday night.

Note: Naturally you also need to have the script installed on the servers you are ssh’ing into

precmd() hook

You can run the script once and the tab color is set. However, if you SSH from the computer to another  and then exit back, the color of the latest server would remain in this case.

This can be avoided by

1. Calculating the OSC control code sequence needed to set the terminal tab color when the shell starts
2. Have a precmd() hook (zsh terminology, not sure what other shells use) to reset the tab color every time the shell prompt is displayd

We, me with my friend, are maintaining (yet another) zsh toolkit called ztanesh (github). There  you can find precmd() example codes in 1) 98-server-color and 2) 80-statusbar.

rainbow-parade.py

The script code lives on Github. Currently it supports iTerm 2 only and we plan to expand support to Konsole. Patches for other terminals are welcome.

(This probably could be done in pure shell code too, but Python is just so much more fun…)

#!/usr/bin/env python
"""

       Set terminal tab / decoration color by the server name.

       Get a random colour which matches the server name and use it for the tab colour:
       the benefit is that each server gets a distinct color which you do not need
       to configure beforehand.

"""

import socket
import random
import colorsys
import sys

# http://stackoverflow.com/questions/1523427/python-what-is-the-common-header-format
__copyright__ = "Copyright 2012 Mikko Ohtamaa - http://opensourcehacker.com"
__author__ = "Mikko Ohtamaa <mikko@opensourcehacker.com>"
__licence__ = "WTFPL"
__credits__ = ["Antti Haapala"]

USAGE = """
Colorize terminal tab based on the current host name.

Usage: rainbow-parade.py [0-1.0] [0-1.0] # Lightness and saturation values

An iTerm 2 example (recolorize dark grey background and black text):

    rainbow-parade.py 0.7 0.4
"""

def get_random_by_string(s):
    """
    Get always the same 0...1 random number based on an arbitrary string
    """

    # Initialize random gen by server name hash
    random.seed(s)
    return random.random()

def decorate_terminal(color):
    """
    Set terminal tab / decoration color.

    Please note that iTerm 2 / Konsole have different control codes over this.
    Note sure what other terminals support this behavior.

    :param color: tuple of (r, g, b)
    """

    r, g, b = color

    # iTerm 2
    # http://www.iterm2.com/#/section/documentation/escape_codes"
    sys.stdout.write("\033]6;1;bg;red;brightness;%d\a" % int(r * 255))
    sys.stdout.write("\033]6;1;bg;green;brightness;%d\a" % int(g * 255))
    sys.stdout.write("\033]6;1;bg;blue;brightness;%d\a" % int(b * 255))
    sys.stdout.flush()

    # Konsole
    # TODO
    # http://meta.ath0.com/2006/05/24/unix-shell-games-with-kde/

def rainbow_unicorn(lightness, saturation):
    """
    Colorize terminal tab by your server name.

    Create a color in HSL space where lightness and saturation is locked, tune only hue by the server.

    http://games.adultswim.com/robot-unicorn-attack-twitchy-online-game.html
    """

    name = socket.gethostname()

    hue = get_random_by_string(name)

    color = colorsys.hls_to_rgb(hue, lightness, saturation)

    decorate_terminal(color)

def main():
    """
    From Toholampi with love http://www.toholampi.fi/tiedostot/119_yleisesite_englanti_naytto.pdf
    """
    if(len(sys.argv) < 3):
        sys.exit(USAGE)

    lightness = float(sys.argv[1])
    saturation = float(sys.argv[2])

    rainbow_unicorn(lightness, saturation)

if __name__ == "__main__":
    main()

 Subscribe to this blog in a reader Follow me on Twitter

1. You can get genuinely high quality code hosting for free. Sure Sourceforge was already around back then, but Git and Mercurial stomp all over CVS from a collaboration point of view. These also come with decent issue trackers and various other collaboration tools. If you don't want to trust a service provider with your code, than tools like GitLab let you set up similar environments internally.
2. Web based issue trackers are everywhere, with the ubiquitous "issue URL" allowing effective cross-linking between tracker issues, documentation, code comments, source control browsers, code review systems, etc.
3. Dedicated code review tools like Gerrit and Reitveld are published as open source (and, in the case of the latter, even available as a free service on Google App Engine).
4. Services like ReadTheDocs exist, allowing you to easily build and publish high quality documentation. All with nice URLs so you can link it from emails, tracker issues, source code, etc.
5. Organisations like Shining Panda CI and Travis CI provide hosted continuous integration services that put the internal capabilities of many large companies to shame.
6. Language communities provide cross-platform distribution services to reach a global audience.
7. Depending on the language you use, you may even have tools like SonarSource available
8. Once you go into production in the web application world, service components like Sentry, Piwik, and Graphite are again available for no charge.

• they're hosted on BitBucket as Mercurial projects (I happen to prefer Mercurial, although I can definitely see why people like Git, too). That gives me integrated issue tracking and online source code browsing, too. (OK, so I could have had essentially that back in the early SourceForge days, but the UI aspects have improved in many respects in the intervening years)
• I can publish my projects on the Python Package Index with a simple "setup.py sdist upload". They're then available for anyone in the world to install with a straightforward command like "pip install walkdir"
• thanks to Shining Panda CI, I know the downloads from PyPI work, and I also know that the projects work on all the versions and implementations of Python I want to support
• thanks to ReadTheDocs and Sphinx, you can read nicely formatted documentation like this rather than trying to decipher plain text files or wiki pages.

Enough of my jibber-jabber. Here is my Haskell solution first (trying to change things up here):

module Main where
 
main :: IO()
main = do
    print . take 10 . show $ sum big_number
    where big_number = [ 37107287533902102798797998220837590246510135740250
                       , 46376937677490009712648124896970078050417018260538
                       , 74324986199524741059474233309513058123726617309629
                       , 91942213363574161572522430563301811072406154908250
                       , 23067588207539346171171980310421047513778063246676
                       , 89261670696623633820136378418383684178734361726757
                       , 28112879812849979408065481931592621691275889832738
                       , 44274228917432520321923589422876796487670272189318
                       , 47451445736001306439091167216856844588711603153276
                       , 70386486105843025439939619828917593665686757934951
                       , 62176457141856560629502157223196586755079324193331
                       , 64906352462741904929101432445813822663347944758178
                       , 92575867718337217661963751590579239728245598838407
                       , 58203565325359399008402633568948830189458628227828
                       , 80181199384826282014278194139940567587151170094390
                       , 35398664372827112653829987240784473053190104293586
                       , 86515506006295864861532075273371959191420517255829
                       , 71693888707715466499115593487603532921714970056938
                       , 54370070576826684624621495650076471787294438377604
                       , 53282654108756828443191190634694037855217779295145
                       , 36123272525000296071075082563815656710885258350721
                       , 45876576172410976447339110607218265236877223636045
                       , 17423706905851860660448207621209813287860733969412
                       , 81142660418086830619328460811191061556940512689692
                       , 51934325451728388641918047049293215058642563049483
                       , 62467221648435076201727918039944693004732956340691
                       , 15732444386908125794514089057706229429197107928209
                       , 55037687525678773091862540744969844508330393682126
                       , 18336384825330154686196124348767681297534375946515
                       , 80386287592878490201521685554828717201219257766954
                       , 78182833757993103614740356856449095527097864797581
                       , 16726320100436897842553539920931837441497806860984
                       , 48403098129077791799088218795327364475675590848030
                       , 87086987551392711854517078544161852424320693150332
                       , 59959406895756536782107074926966537676326235447210
                       , 69793950679652694742597709739166693763042633987085
                       , 41052684708299085211399427365734116182760315001271
                       , 65378607361501080857009149939512557028198746004375
                       , 35829035317434717326932123578154982629742552737307
                       , 94953759765105305946966067683156574377167401875275
                       , 88902802571733229619176668713819931811048770190271
                       , 25267680276078003013678680992525463401061632866526
                       , 36270218540497705585629946580636237993140746255962
                       , 24074486908231174977792365466257246923322810917141
                       , 91430288197103288597806669760892938638285025333403
                       , 34413065578016127815921815005561868836468420090470
                       , 23053081172816430487623791969842487255036638784583
                       , 11487696932154902810424020138335124462181441773470
                       , 63783299490636259666498587618221225225512486764533
                       , 67720186971698544312419572409913959008952310058822
                       , 95548255300263520781532296796249481641953868218774
                       , 76085327132285723110424803456124867697064507995236
                       , 37774242535411291684276865538926205024910326572967
                       , 23701913275725675285653248258265463092207058596522
                       , 29798860272258331913126375147341994889534765745501
                       , 18495701454879288984856827726077713721403798879715
                       , 38298203783031473527721580348144513491373226651381
                       , 34829543829199918180278916522431027392251122869539
                       , 40957953066405232632538044100059654939159879593635
                       , 29746152185502371307642255121183693803580388584903
                       , 41698116222072977186158236678424689157993532961922
                       , 62467957194401269043877107275048102390895523597457
                       , 23189706772547915061505504953922979530901129967519
                       , 86188088225875314529584099251203829009407770775672
                       , 11306739708304724483816533873502340845647058077308
                       , 82959174767140363198008187129011875491310547126581
                       , 97623331044818386269515456334926366572897563400500
                       , 42846280183517070527831839425882145521227251250327
                       , 55121603546981200581762165212827652751691296897789
                       , 32238195734329339946437501907836945765883352399886
                       , 75506164965184775180738168837861091527357929701337
                       , 62177842752192623401942399639168044983993173312731
                       , 32924185707147349566916674687634660915035914677504
                       , 99518671430235219628894890102423325116913619626622
                       , 73267460800591547471830798392868535206946944540724
                       , 76841822524674417161514036427982273348055556214818
                       , 97142617910342598647204516893989422179826088076852
                       , 87783646182799346313767754307809363333018982642090
                       , 10848802521674670883215120185883543223812876952786
                       , 71329612474782464538636993009049310363619763878039
                       , 62184073572399794223406235393808339651327408011116
                       , 66627891981488087797941876876144230030984490851411
                       , 60661826293682836764744779239180335110989069790714
                       , 85786944089552990653640447425576083659976645795096
                       , 66024396409905389607120198219976047599490197230297
                       , 64913982680032973156037120041377903785566085089252
                       , 16730939319872750275468906903707539413042652315011
                       , 94809377245048795150954100921645863754710598436791
                       , 78639167021187492431995700641917969777599028300699
                       , 15368713711936614952811305876380278410754449733078
                       , 40789923115535562561142322423255033685442488917353
                       , 44889911501440648020369068063960672322193204149535
                       , 41503128880339536053299340368006977710650566631954
                       , 81234880673210146739058568557934581403627822703280
                       , 82616570773948327592232845941706525094512325230608
                       , 22918802058777319719839450180888072429661980811197
                       , 77158542502016545090413245809786882778948721859617
                       , 72107838435069186155435662884062257473692284509516
                       , 20849603980134001723930671666823555245252804609722
                       , 53503534226472524250874054075591789781264330331690]

followed by my Python solution:

#!/usr/bin/python
"""
code solution for project euler's problem #13 in python.
"""
from __future__ import print_function
 
def print_10(number):
    print(str(number)[0:10])
 
if __name__ == "__main__":
 
    big_number = [ 37107287533902102798797998220837590246510135740250,
                   46376937677490009712648124896970078050417018260538,
                   74324986199524741059474233309513058123726617309629,
                   91942213363574161572522430563301811072406154908250,
                   23067588207539346171171980310421047513778063246676,
                   89261670696623633820136378418383684178734361726757,
                   28112879812849979408065481931592621691275889832738,
                   44274228917432520321923589422876796487670272189318,
                   47451445736001306439091167216856844588711603153276,
                   70386486105843025439939619828917593665686757934951,
                   62176457141856560629502157223196586755079324193331,
                   64906352462741904929101432445813822663347944758178,
                   92575867718337217661963751590579239728245598838407,
                   58203565325359399008402633568948830189458628227828,
                   80181199384826282014278194139940567587151170094390,
                   35398664372827112653829987240784473053190104293586,
                   86515506006295864861532075273371959191420517255829,
                   71693888707715466499115593487603532921714970056938,
                   54370070576826684624621495650076471787294438377604,
                   53282654108756828443191190634694037855217779295145,
                   36123272525000296071075082563815656710885258350721,
                   45876576172410976447339110607218265236877223636045,
                   17423706905851860660448207621209813287860733969412,
                   81142660418086830619328460811191061556940512689692,
                   51934325451728388641918047049293215058642563049483,
                   62467221648435076201727918039944693004732956340691,
                   15732444386908125794514089057706229429197107928209,
                   55037687525678773091862540744969844508330393682126,
                   18336384825330154686196124348767681297534375946515,
                   80386287592878490201521685554828717201219257766954,
                   78182833757993103614740356856449095527097864797581,
                   16726320100436897842553539920931837441497806860984,
                   48403098129077791799088218795327364475675590848030,
                   87086987551392711854517078544161852424320693150332,
                   59959406895756536782107074926966537676326235447210,
                   69793950679652694742597709739166693763042633987085,
                   41052684708299085211399427365734116182760315001271,
                   65378607361501080857009149939512557028198746004375,
                   35829035317434717326932123578154982629742552737307,
                   94953759765105305946966067683156574377167401875275,
                   88902802571733229619176668713819931811048770190271,
                   25267680276078003013678680992525463401061632866526,
                   36270218540497705585629946580636237993140746255962,
                   24074486908231174977792365466257246923322810917141,
                   91430288197103288597806669760892938638285025333403,
                   34413065578016127815921815005561868836468420090470,
                   23053081172816430487623791969842487255036638784583,
                   11487696932154902810424020138335124462181441773470,
                   63783299490636259666498587618221225225512486764533,
                   67720186971698544312419572409913959008952310058822,
                   95548255300263520781532296796249481641953868218774,
                   76085327132285723110424803456124867697064507995236,
                   37774242535411291684276865538926205024910326572967,
                   23701913275725675285653248258265463092207058596522,
                   29798860272258331913126375147341994889534765745501,
                   18495701454879288984856827726077713721403798879715,
                   38298203783031473527721580348144513491373226651381,
                   34829543829199918180278916522431027392251122869539,
                   40957953066405232632538044100059654939159879593635,
                   29746152185502371307642255121183693803580388584903,
                   41698116222072977186158236678424689157993532961922,
                   62467957194401269043877107275048102390895523597457,
                   23189706772547915061505504953922979530901129967519,
                   86188088225875314529584099251203829009407770775672,
                   11306739708304724483816533873502340845647058077308,
                   82959174767140363198008187129011875491310547126581,
                   97623331044818386269515456334926366572897563400500,
                   42846280183517070527831839425882145521227251250327,
                   55121603546981200581762165212827652751691296897789,
                   32238195734329339946437501907836945765883352399886,
                   75506164965184775180738168837861091527357929701337,
                   62177842752192623401942399639168044983993173312731,
                   32924185707147349566916674687634660915035914677504,
                   99518671430235219628894890102423325116913619626622,
                   73267460800591547471830798392868535206946944540724,
                   76841822524674417161514036427982273348055556214818,
                   97142617910342598647204516893989422179826088076852,
                   87783646182799346313767754307809363333018982642090,
                   10848802521674670883215120185883543223812876952786,
                   71329612474782464538636993009049310363619763878039,
                   62184073572399794223406235393808339651327408011116,
                   66627891981488087797941876876144230030984490851411,
                   60661826293682836764744779239180335110989069790714,
                   85786944089552990653640447425576083659976645795096,
                   66024396409905389607120198219976047599490197230297,
                   64913982680032973156037120041377903785566085089252,
                   16730939319872750275468906903707539413042652315011,
                   94809377245048795150954100921645863754710598436791,
                   78639167021187492431995700641917969777599028300699,
                   15368713711936614952811305876380278410754449733078,
                   40789923115535562561142322423255033685442488917353,
                   44889911501440648020369068063960672322193204149535,
                   41503128880339536053299340368006977710650566631954,
                   81234880673210146739058568557934581403627822703280,
                   82616570773948327592232845941706525094512325230608,
                   22918802058777319719839450180888072429661980811197,
                   77158542502016545090413245809786882778948721859617,
                   72107838435069186155435662884062257473692284509516,
                   20849603980134001723930671666823555245252804609722,
                   53503534226472524250874054075591789781264330331690]
 
    print_10(sum(big_number))

and to continue adding in the spice, I have included a solution in Scala:

import BigInt._
 
object problem_13 {
    def main (args : Array[String]){
        val big_number = List("37107287533902102798797998220837590246510135740250",
                              "46376937677490009712648124896970078050417018260538",
                              "74324986199524741059474233309513058123726617309629",
                              "91942213363574161572522430563301811072406154908250",
                              "23067588207539346171171980310421047513778063246676",
                              "89261670696623633820136378418383684178734361726757",
                              "28112879812849979408065481931592621691275889832738",
                              "44274228917432520321923589422876796487670272189318",
                              "47451445736001306439091167216856844588711603153276",
                              "70386486105843025439939619828917593665686757934951",
                              "62176457141856560629502157223196586755079324193331",
                              "64906352462741904929101432445813822663347944758178",
                              "92575867718337217661963751590579239728245598838407",
                              "58203565325359399008402633568948830189458628227828",
                              "80181199384826282014278194139940567587151170094390",
                              "35398664372827112653829987240784473053190104293586",
                              "86515506006295864861532075273371959191420517255829",
                              "71693888707715466499115593487603532921714970056938",
                              "54370070576826684624621495650076471787294438377604",
                              "53282654108756828443191190634694037855217779295145",
                              "36123272525000296071075082563815656710885258350721",
                              "45876576172410976447339110607218265236877223636045",
                              "17423706905851860660448207621209813287860733969412",
                              "81142660418086830619328460811191061556940512689692",
                              "51934325451728388641918047049293215058642563049483",
                              "62467221648435076201727918039944693004732956340691",
                              "15732444386908125794514089057706229429197107928209",
                              "55037687525678773091862540744969844508330393682126",
                              "18336384825330154686196124348767681297534375946515",
                              "80386287592878490201521685554828717201219257766954",
                              "78182833757993103614740356856449095527097864797581",
                              "16726320100436897842553539920931837441497806860984",
                              "48403098129077791799088218795327364475675590848030",
                              "87086987551392711854517078544161852424320693150332",
                              "59959406895756536782107074926966537676326235447210",
                              "69793950679652694742597709739166693763042633987085",
                              "41052684708299085211399427365734116182760315001271",
                              "65378607361501080857009149939512557028198746004375",
                              "35829035317434717326932123578154982629742552737307",
                              "94953759765105305946966067683156574377167401875275",
                              "88902802571733229619176668713819931811048770190271",
                              "25267680276078003013678680992525463401061632866526",
                              "36270218540497705585629946580636237993140746255962",
                              "24074486908231174977792365466257246923322810917141",
                              "91430288197103288597806669760892938638285025333403",
                              "34413065578016127815921815005561868836468420090470",
                              "23053081172816430487623791969842487255036638784583",
                              "11487696932154902810424020138335124462181441773470",
                              "63783299490636259666498587618221225225512486764533",
                              "67720186971698544312419572409913959008952310058822",
                              "95548255300263520781532296796249481641953868218774",
                              "76085327132285723110424803456124867697064507995236",
                              "37774242535411291684276865538926205024910326572967",
                              "23701913275725675285653248258265463092207058596522",
                              "29798860272258331913126375147341994889534765745501",
                              "18495701454879288984856827726077713721403798879715",
                              "38298203783031473527721580348144513491373226651381",
                              "34829543829199918180278916522431027392251122869539",
                              "40957953066405232632538044100059654939159879593635",
                              "29746152185502371307642255121183693803580388584903",
                              "41698116222072977186158236678424689157993532961922",
                              "62467957194401269043877107275048102390895523597457",
                              "23189706772547915061505504953922979530901129967519",
                              "86188088225875314529584099251203829009407770775672",
                              "11306739708304724483816533873502340845647058077308",
                              "82959174767140363198008187129011875491310547126581",
                              "97623331044818386269515456334926366572897563400500",
                              "42846280183517070527831839425882145521227251250327",
                              "55121603546981200581762165212827652751691296897789",
                              "32238195734329339946437501907836945765883352399886",
                              "75506164965184775180738168837861091527357929701337",
                              "62177842752192623401942399639168044983993173312731",
                              "32924185707147349566916674687634660915035914677504",
                              "99518671430235219628894890102423325116913619626622",
                              "73267460800591547471830798392868535206946944540724",
                              "76841822524674417161514036427982273348055556214818",
                              "97142617910342598647204516893989422179826088076852",
                              "87783646182799346313767754307809363333018982642090",
                              "10848802521674670883215120185883543223812876952786",
                              "71329612474782464538636993009049310363619763878039",
                              "62184073572399794223406235393808339651327408011116",
                              "66627891981488087797941876876144230030984490851411",
                              "60661826293682836764744779239180335110989069790714",
                              "85786944089552990653640447425576083659976645795096",
                              "66024396409905389607120198219976047599490197230297",
                              "64913982680032973156037120041377903785566085089252",
                              "16730939319872750275468906903707539413042652315011",
                              "94809377245048795150954100921645863754710598436791",
                              "78639167021187492431995700641917969777599028300699",
                              "15368713711936614952811305876380278410754449733078",
                              "40789923115535562561142322423255033685442488917353",
                              "44889911501440648020369068063960672322193204149535",
                              "41503128880339536053299340368006977710650566631954",
                              "81234880673210146739058568557934581403627822703280",
                              "82616570773948327592232845941706525094512325230608",
                              "22918802058777319719839450180888072429661980811197",
                              "77158542502016545090413245809786882778948721859617",
                              "72107838435069186155435662884062257473692284509516",
                              "20849603980134001723930671666823555245252804609722",
                              "53503534226472524250874054075591789781264330331690") map {BigInt(_)}
        val sums = big_number sum
        val su = sums toString
        val su10 = su take 10
        println(su10)
    }
}

The solution, in all three languages, is pretty simple. The recipe essentially says, “Put all numbers into a list. Get the sum of that list, turn that number into a string, and get the first 10 characters of that string.”

Times:
Haskell (compiled) : real 0m0.004s
Haskell (runghc) : real 0m0.314s
Python : real 0m0.059s
Scala (compiled) : real 0m0.757s

For the most part it's pretty standard in these tests to see performance times such that Haskell (compiled) Python Haskell (runghc). Java and Perl usually fall somewhere between the Haskell (compiled) and Python, in that order. To see Scala be 2x slower than Haskell (runghc) was a shocker. The only thing that makes sense to me for the slowdown is having to use the BigInt library. That is probably the biggest thing I took away from these time tests - if I want to do REALLY large number crunching and performance DOES matter, JVM-based languages might not be the best option.

A few thoughts on Scala:
If I haven't stated it already in this blog, I should now give the disclaimer that I'm not a Java fan. I know it still has its loyal followers, but I'm not one of them. Moving on. This was my first time working with Scala, and I'd like to finally welcome Java to the 21st century. While doing some research on the Scala language itself I read that “the industry” was moving to replace Java with Scala. I welcome that change. Does that mean I “like” Scala? The honest answer is, to butcher the quote the appliances from the Flintstones, “Eh, it's a language.” Scala is definitely an improvement over Java – not really that hard to do in my opinion – but, the language still feels unpolished. One quick way to kill the interpreter in Scala is to type “Int” then hit the enter key. Instead of error-ing out, the interpreter does a great job of interpreting a crash test car hitting a cement wall (I had to restart the whole thing.) When I tried the same “technique” in the Python interpreter, I got as a response and for Haskell's interpreter I received “Not in scope: data constructor `Int'”. I also found Scala's function composition to be a little lacking when compared to Haskell. I wasn't able to cleanly change the BigInt data type to String, and then only print out ten characters without requiring three separate val's. Yes, I could have used one var instead, but that's beside the point. I will admit it could be my inexperience with the language showing, so if anyone knows a smoother way to do this in Scala please share it in the comments.

All that being said, I do like the way Scala is trying to handle the reducing of Java's dot notation, and I think it's starting to make strides in the right direction in other areas. I'm open to working with Scala more, and look forward to seeing how it evolves over the next few years.

This version requires the following packages pip installed into your virtualenv.

• django-crispy-forms so we can do Python driven layouts.
• django-floppyforms so we get HTML5 elements for free.

They also need to be added to your list of INSTALLED_APPS:

INSTALLED_APPS += (
    'crispy_forms',
    'floppyforms',
)

In myapp.forms.py:

from crispy_forms.helper import FormHelper
from crispy_forms.layout import Submit
import floppyforms as forms

class ContactForm(forms.Form):

    name = forms.CharField(required=True)
    email = forms.EmailField(required=True)
    subject = forms.CharField(required=True)
    message = forms.CharField(widget=forms.Textarea)

    def __init__(self, *args, **kwargs):
        self.helper = FormHelper()
        self.helper.add_input(Submit('submit', 'Submit'))
        super(ContactForm, self).__init__(*args, **kwargs)

In myapp.views.py:

from django.conf import settings
from django.core.mail import send_mail
from django.views.generic import FormView

from myapp.forms import ContactForm

class ContactFormView(FormView):

    form_class = ContactForm
    template_name = "myapp/email_form.html"
    success_url = '/email-sent/'

    def form_valid(self, form):
        message = "{name} / {email} said: ".format(
            name=form.cleaned_data.get('name'),
            email=form.cleaned_data.get('email'))
        message += "\n\n{0}".format(form.cleaned_data.get('message'))
        send_mail(
            subject=form.cleaned_data.get('subject'),
            message=message,
            from_email='contact-form@myapp.com',
            recipient_list=[settings.LIST_OF_EMAIL_RECIPIENTS],
        )
        return super(ContactFormView, self).form_valid(form)

In templates/myapp/email_form.html:

{% extends 'base.html' %}
{% load crispy_forms_tags %}

{% block title %}Send an email{% endblock %}

{% block content %}
    <div class="row">
        <div class="span6">
            <h1>Send an email</h1>
            {% crispy form form.helper %}
        </div>
    </div>
{% endblock %}

{% block extrajs %}
<script src="{{ STATIC_URL }}js/jquery-1.7.1.min.js"></script>
<script type="text/javascript">
$(function() {
    $('#id_name').focus()
});
</script>
{% endblock %}