I do not claim to have achieved any kind of proficiency in Haskell, of course; I might very well be just scratching the surface. However, this is exactly the sort of perspective I wanted to employ when evaluating language usefulness: a practical standpoint, held by programmer who is looking to use it for actual tasks, without having mastered it in great detail - at least initially. This is, by the way, a pretty common setting when tackling all new things related to coding, be it frameworks, software platforms or languages.

Still, I knew it was almost supposed to be a rough ride. A language whose tutorials purposefully shy away from classic Hello World example (typically inserting a factorial function instead) looks like something designed specifically to melt brains of poor programmers who dare to venture forth to investigate it. Those few who make their way back are told about in folk tales, portrayed as mildly crazy types who profess this weird idea of "purity", and utter the word 'monad' with great contempt.

Okay, I may be exaggerating... slightly. But there is no denying that Haskell attained a certain kind of reputation, something like a quirky cousin in the big and diverse family of languages. Unlike Lisp, he isn't picked on due to any (particular (and (rather) obvious)) property. There is just this general aura of weirdness and impracticality that he supposedly radiates, repelling all but the most adventurous of coders.

If it really exists, then pity: it certainly failed to repel me. As a result, I may have a story (or two) to share with those who want to learn what's this functional business is really about, and why should they care.

Grab a chair and something to drink - it's going to take a while. But in the end, you shouldn't regret staying.

Introduction of main protagonist

Regardless of what prior knowledge you may already have, it is certainly useful to begin with the basics.

Haskell Curry (1900-1982)

Haskell is a general purpose, compiled, statically (and strongly) typed, purely functional programming language with lazy evaluation, expansive and modular standard library, a rich repository of external packages, support for all major operating systems via the GHC compiler, and peculiar ability of making blogposts sound like Wikipedia articles. It doesn't seem to have a single creator (in a way that, for instance, Python has Guido von Rossum), but its namesake is Haskell Curry, a 20th century mathematician and logician.

As for popularity, it currently sports a not-very-impressive 42th place on the Tiobe Index - notably above Go, Clojure, Dart and Scala, and unsurprisingly below most languages you would think about. GitHub currently hosts around 6700 repositories whose projects use Haskell in any way - compared to, say, about 80,000 for Python, 40,000 for C and 160,000 for Ruby (yikes!). A little below 6000 questions tagged haskell has been asked on StackOverflow, putting this tag on the same page as boost, jquery-plugins, version-control and soap (eww!).

Motivation

So it goes without saying that Haskell seems to be rather hip..., ahem, niche language. In the programming world, this should be considered a liability, since larger community around a language naturally results in more third-party libraries, open source projects and yes, job offers. On the other hand, we know the old saying about quantity triumphing quality, which we are inclined to believe at least in some cases (*cough* Java).

If not popularity, then what are the Haskell's immediately recognizable treats that could appeal to newcomers?... For one, it's the GHC package, which stands for Glasgow Haskell Compiler. Easily available on any desktop platform, it's not only a compiler but also an interpreter which can run scripts (via runghc command), and act as a REPL (Read-Eval-Print Loop) for quick experiments (via ghci). That last thing is especially important, as I've grown to perceive languages that refuse to be poked at with their own shell as just plain barbaric. Haskell proves it's entirely possible for a compiled language to offer sensible, efficient and useful REPL.

Beware of big concepts in small pieces

What about writing real code? As for actual programs, they are organized into modules (.hs files) which form a nice hierarchy of namespaces, just like you would expect when coming from Java, .NET, etc. One neat thing here is how external packages fit into the whole picture. Rather than rolling out an independent root namespace for every custom package, convention orders to place third-party modules within existing tree. As a result, stuff related to networking goes always into Network namespace, text utilities (e.g. parsers) into Text, data structures into Data, OS stuff into System, and so on.

On more local level, the most basic unit of code structure is of course a function. Haskell functions are slightly different than usual, because they are required to be referentially transparent: they must always return the same value for the same arguments. Also, they cannot have the so-called side effects, which excludes pretty much anything outside of actual computation - most notably I/O.
How this can be reconciled with being able to, well, do anything practical shall be left for now. (Some may know that it's about the infamous M-word). What I'd like to point out instead is that a function in Haskell is just a name for (parametrized) expression. If you're a Python programmer, you can imagine functions being always defined with lambda rather than typical def. Because there really isn't much you can cram into a single expression, you are basically required to aggressively split things into smaller and smaller bits, using let-in or where syntax. Here's a simple example of such splitting, done for a function that is already pretty trivial - it just formats an URL:

Hopefully I don't have to elaborate overmuch on how this is a good thing, to have your code organized into many small, manageable parts that clearly relate to each other. In Haskell, this is almost the only way you can actually produce anything, which is both marvelous and scary at the same time. There are simply no excuses for quick & dirty hacks: refactoring happens while you are coding, not afterwards.

Types of types

Besides algorithms (functions), we know that programs are composed of data structures, and in case of Haskell they translate into data types. Quite amusingly, if we don't count aliases, there is just one sort of types that can be defined here. It's called algebraic data type (ADT) and it's entirely sufficient, because it simply has everything. And by that I mean literally everything, because it allows to define stuff like enums:

as well as structures (records):

and unions:

Not only that - it also permits types to have parameters (with constraints):

which easily satisfies the definition of 'template', 'generic type' and 'concept' from other languages. And if you haven't noticed, we just combined half of semantics of C and the most powerful feature of C++ into one syntactical construct - and rather simple one, too. I don't know about you, but for me it just oozes awesomeness.

Really strong typing

The perks of Haskell type system are not limited to ADTs, though. As I have mentioned earlier, Haskell is a compiled and statically typed language. For many programmers this may bring immediate associations with cryptic error messages and pointless things littering the code just to please the compiler. In reality, static typing was always meant as a tool to reinforce code correctness; but often, it just didn't seem to bring enough benefits to justify the headaches involved.

Haskell appears to be offering a proper balance. Because it's very good at figuring out the types by itself (via type inference), it seldom requires explicit type declarations. (They are still recommended for top-level functions, though). And since the type system is extremely powerful, it can often serve as a sufficient proof of program's correctness. Basically: if it compiles, it has pretty high chance of working - much higher than in other statically typed languages, at least.

To be honest, this happens not only due to elaborate mixing and matching of types, but also because of referential transparency. Constraining the class of possible actions that a particular piece of code can undertake goes a long way towards preventing errors. Furthermore, it greatly simplifies the testing process, up to the point of rendering many typical kinds of unit tests obsolete. Indeed, Haskell's testing facility known as QuickCheck can generate test cases automatically, based on a set of properties that we expect our functions to hold.

You don't think FP-dimensionally

Going through all this features and learning about those wonders of modern technology, you might be left thinking: where's the trick here?... It's wise to not let yourself be caught off-guard, for indeed there are some downsides. Several of them, actually.

An obvious one seems to be the functional paradigm itself, especially if coming from a language that wasn't significantly influenced by FP concepts. However, the pool of languages that still lack at least basic functional features is steadily shrinking. A relatively recent example could be C++, which has just gained a superior support for closures and functions as first-order values, supplementing its already existing capabilities. Others - like popular scripting languages - are usually going even further in their support for functional style.

So if you have done some serious coding in any of those languages, you had good chance of seeing the elements of FP in action. Heck, there is even a non-zero probability that you have used them, which should definitely help with cushioning the Haskell's blow. And if structuring your code using maps, folds, pattern matching and guards turns out to be insufficient, there is always a place for plain ol' loops. Sure, they are called "tail recursion" and may not have an obvious 1:1 correspondence to iterative constructs from imperative languages, but they are close enough to work in every reasonable case - if only somewhat inelegantly.

Which style do you choose?

Unfortunately, getting past the initial obstacle and learning to frame your thoughts into purely functional code is just a start. See, on some level Haskell can be viewed as conglomerate of several sublanguages, defined by functions and operators applicable to specific typeclasses. A typeclass is rather simple concept here; it can be explained as slightly more powerful version of an interface from traditional OOP. Concrete types can be instances of one or more typeclass, just like "normal" classes are said to be implementing interfaces.

In Haskell, however, some typeclasses are so influential that their operators and functions essentially dictate the overall shape of a piece of code. Therefore, you can often hear about few different "styles" of Haskell code, and the most prominent ones seem to be: applicative, arrow-based and monadic style.

When writing any more complicated piece of logic, you are generally required to decide early on which style you are going to use. The choice is often non-obvious, as the foundational bases behind different styles are closely related, often through inclusion (like applicative functors and monads). Each style has a distinct "feel" to it, though, and correspondingly, it looks noticeably different in actual code. And while it's entirely possible to mix several styles together, it requires great care if result is to be in any way comprehensible later on.

It's not hard; it's abstract

At this point it's quite natural to ask: what are those things anyway, those functors, arrows and *deep breath* monads?... Actually, I could add a few more to the mix. Meet categories, foldables, traversables, monad transformers and monoids; I'm sure it's a beginning of a long-lasting friendship!

Jokes aside: the answer is both simple and complicated. These are abstract concepts, originating mostly from category theory and pertaining to either computational processes, or entities that those processes are operating on. Note that by 'abstract', I mean real, mathematical abstraction - not the flamboyant, object-oriented stuff that's making wimpy coders shudder because they've seen too many service provider factories. That's just flexibility at best, and complexity at worst - but nothing there requires cracking big, mental roadblocks in order to understand.

With terms I've listed above, it's entirely different situation. Sure, they are all built from quite simple, elementary building blocks: functions, their combinations and parametrized data types. But somewhere along the way up, there is always an initial mental gap which has to be filled with our own, individual analogies that allow us to grasp these abstract concepts. Those analogies are likely specific to given individual and might very well be impossible to put into words. Because of that, there is no royal road to enlightenment: one must simply bash their head against those ideas long enough for those helpful analogies to finally emerge.

Oh, and don't think you'll be able to communicate them to others. It's been tried enough times as to attain a dedicated term: the monad tutorial fallacy.

parseJSON :: Result Headache

You may argue - and be absolutely correct - that this situation isn't really specific to Haskell or functional programming. Indeed, it happens when we start to tackle any new programming paradigm, and especially when just learning to code for the first time. But many of us simply aren't used to be at that level anymore, especially if their last time was few years ago or even further in the past. Thus, in a sense, taking up Haskell often begins with a lesson in humility. Hardly a pleasant experience, but apparently a necessary one.

I would be totally fine with intricate semantics, though, if things that are obviously fixable weren't getting in the way so very often. One of those things is the miserable excuse for "documentation" that many Haskell packages are being shipped with. And I mean packages with absolutely crucial functionality, such as regular expressions' library or - my personal nemesis - JSON parser package. In the latter case, the documentation is so overwhelmingly useless that I had to spend better part of an hour just to figure out any kind of starting point to work my way from. Then, fleshing out the details for a final parsing logic of a a very simple project took more than one evening.

To emphasize: I'm talking about handling JSON here, a bread and butter of most contemporary applications that have anything to do with the Internet. This is something that should be as trivial as putting a string together, or sorting a collection via custom comparator. It's not like there aren't any precedents here, showing that it is entirely doable, and thus a reasonable thing to request from a language or its standard library. Yet it is apparently not the case in Haskell, which is just shameful and sad.

How much magic can you take?

Naturally, this isn't something that cannot be amended, and I hope for Haskell packages' docs to steadily improve over time. Besides, hiccups like that are not nearly enough to dispel the irrefutable "magic" that Haskell emanates - although admittedly, not every shade of it is something to be satisfied with either.

Surely, there's plenty of magic of the Arthur C. Clarke type. Some of this good stuff I briefly touched upon in the beginning, but there's a lot more things worthy of attention here. One of those may be an out-of-the-box concurrency, which allows to automatically parallelize computing of pure code. Combined with support for software transactional memory (STM), it can turn hard problems of concurrent programming into something much more manageable.

But unfortunately, it seems that the other kind of coding magic - of bad type - is also abundant in Haskell. It's the magic of doing a whole lot of stuff behind the scenes, out of programmer's sight. It is present in the very idea of lazy evaluation, resulting in detachment between the place where expression is put in code and the actual moment where it is computed. This may sometimes result in unwanted performance implications, forcing us to use seq or bangs (e.g. $! for function application) in order to regain some control over evaluation.

Hidden magic can also lie in syntax, obscuring the details of what particular piece of code really does. Sometimes it might even allow for one snippet to do several totally different things, as if you had overloaded the semicolon "operator" in C-like language. 'Programmable semicolons' is actually one of the many epithets associated with monads, and the reason for that can be easily seen in the following function f:

If we execute it within context of IO monad, we will get rather predictable result. (And, as a bonus, we'll finally see that I/O in Haskell is very possible):

But if we use f on two lists, we will get something entirely different:

By some inexplicable sleight of hand, the function has been executed six times, resulting in a Cartesian product of its arguments. For an untrained eye, this has to be really bizarre sort of polymorphism, with no obvious explanation (as I imagine that "list is also a monad" isn't nearly enough).

Eighteen-hole course

You may be blinking now, and wondering: who could have used that in real code, anyway? After all, it's an incredibly effective technique to rapidly decrease the quality of code, as measured by the one and only, true metric. Surely, real haskellers must have much more sanity than that, right?...

By the way, did you notice the names I've used in the preceding samples? Not counting library functions and mandatory main, they were all one-letter names: f, x, y, etc. I'm sure you have thought it's only because this is just an example, unlikely to be replicated in actual code. I must therefore inform you right way: you're greatly mistaken.
Using one-letter identifiers looks like a long-standing Haskell tradition, up to the point of utter confusion. When f can stand for both function or functor (not nearly the same thing); m could be a monoid or monad; t may be traversable, monad transformer or just an arbitrary type - it's hardly surprising to find yourself staring at piece of code in dazing bewilderment.

It won't be only because of cryptic identifiers, though. I noticed that haskellers exhibit a strange fondness to very terse code, dubbing this property "expressiveness". For instance, the following function could be considered relatively typical:

Notice how this function must take an argument (directory's path):

yet there is no identifier that would stand for it in function's declaration or body. It is defined entirely in terms of applying some combinators to functions treated as first-order values, which is often referred to as point-free style, and greatly encouraged. Since there exists plethora of different combinators, this technique can be - and often is - taken really far, exceeding the readability threshold of all but the most seasoned hackers.

This is called "code golf" - a term I've heard for the first time only when diving into Haskell. Frankly, other languages are much more honest when talking about similar practices.

Jumping to conclusions

So, is Haskell beyond salvation?... Hardly. In terms of applications, few areas are already explored very well, with concurrent programs, distributed computing and web programming (yes!) being probably the most important ones. Some others are still relatively pristine, and some can be placed in between: where serious effort is being made, but there aren't yet any universally acclaimed solutions and practices (e.g. programming graphical UIs).

But despite obvious capabilities of performing very practical tasks, Haskell ecosystem as a whole still appears to suffer from the ivory tower syndrome. There is little doubt that learning Haskell is an overall mind-expanding experience, ultimately leading to better code in any language you choose to write in. However, using Haskell for everyday programming tasks is often striding through unknown territory, exploring strange new paradigms, seeking out new techniques and methodologies, to boldly go where no monad has gone before.

All this goodness, however, comes with few strings attached. There are some ways in which we can be bitten by improperly used generators, and I think it's helpful to know about them. So today, I'll try to outline some of those traps.

Generators are lazy

You don't say, eh? That's one of the main reasons we are so keen on using them, after all! But what I'm implying is the following: if a generator-based code is to actually do anything, there must be a point where all this laziness "bottoms out". Otherwise, you are just building expression thunks and not really evaluating anything.

How such circumstances might occur, though? One case involves using generators for their side effects - a rather questionable practice, mind you:

The starmap call does not print anything here, because it just constructs a generator. Only when this generator is iterated over, dictionary elements would be fed to process_kvp function. Such iteration would of course require a loop (or consume recipe from itertools docs), so we might as well do away with generator altogether and just stick with plain old for:

In real code the judgement isn't so obvious, of course. We'll most likely receive the generator from some external function - a one that probably refers to its result only as an iterable. As usual in such cases, we must not assume anything beyond what is explicitly stated. Specifically, we cannot expect that we have any kind of existing, tangible collection with actual elements. An iterable can very well be just a generator, whose items are yet to be evaluated. This fact can impact performance by moving some potentially heavy processing into unexpected places, resulting in e.g. executing database queries while rendering website's template.

Generators are truthy

The above point was concerned mostly with performance, but what about correctness? You may think it's not hard to conform to iterables' contract, which should in turn guarantee that they don't backfire on us with unexpected behavior. Yet how many times did you find yourself reading (or writing!) code such as this:

It's small and may look harmless, but it still manages to make an ungrounded (thus potentially false) assumption about an iterable. The problem lies in if condition, meant to check whether we have any items to do stuff with. It would work correctly if items were a list, dict, deque or any other actual collection. But for generators, it will not work that way - even though they are still undoubtedly iterable!

Why? Because generators are not collections; they are just suppliers. We can only tell them to return their next value, but we cannot peek inside to see if the value is really there. As such, generators are not susceptible to boolean coercion in the same way that collections are; it's not possible to check their "emptiness". They behave like regular Python objects and are always truthy, even if it's "obvious" they won't yield any value when asked:

Going back to previous sample, we can see that if block won't be executed in case of get_stuff_to_do returning an "empty" generator. Consequences of this fact may vary from barely noticeable to disastrous, depending on how the rest of do_stuff function looks like. Nevertheless, that code will run with one of its invariants violated: a fertile ground for any kind of unintended effects.

Generators are once-off

An intuitive, informal understanding of the term 'iterable' is likely to include one more unjustified assumption: that it can iterated over, and over, i.e. multiple times. Again, this is very much true if we're dealing with a collection, but generators simply don't carry enough information to repeat the sequence they yield. In other words, they cannot be rewound: once we go through a generator, it's stuck in its final state, not usable for anything more.

Just like with previous caveats, failure to account for that can very well go unnoticed - at least until we spot some weird behavior it results in. Continuing our superficial example from preceding paragraph, let's pretend the rest of do_stuff function requires going over items at least twice. It could be, for example, an iterable of objects in a game or physics simulation; objects that can potentially move really fast and thus require some more sophisticated collision detection (based on e.g. intersection of segments):

Even assuming the incredible coincidence of getting all the required math right (;-)), we wouldn't see any action whatsoever if items is a generator. The reason for that is simple: when calculate_displacement goes through items once (vigorously applying the Eulerian integration, most likely), it fully expends the generator. For any subsequent traversal (like the one in detect_collitions), the generator will appear empty. In this particular snippet, it will most likely result in blank screen, which hopefully is enough of a hint to figure out what's going on :P

Generators are not lists

An overarching conclusion of the above-mentioned pitfalls is rather evident and seemingly contrary to statement from the beginning. Indeed, generators may not be a drop-in replacement for lists (or other collections) if we are very much relying on their "listy" behavior. And unless memory overhead proves troublesome, it's also not worth it to inject them into older code that already uses lists.

For new code, however, sticking with generators right off the bat has numerous benefits which I mentioned at the start. What it requires, though, is evicting some habits that might have emerged after we spent some time manipulating lists. I think I managed to pinpoint the most common ways in which those habits result in incorrect code. Incidentally, they all origin from accidental, unfounded expectations towards iterables in general. That's no coincidence: generators simply happen to be the "purest" of iterables, supporting only the bare minimum of required operations.

It's not particularly hard, of course. But before jumping to a solution, let's look how we expect our function to behave:

Pretty obvious, isn't it? Now, if you recall how packet routing works under the hood, you might remember that there are some bitwise operations involved. They are necessary to determine whether specific IP address can be found within given network. However low-level this may sound, there is really no escape from doing it this way. Yes, even in Python :P

It goes deeper, though. Most of the interface of Python's socket module is actually carbon-copied from POSIX socket.h and related headers, down to exact names of particular functions. As a result, solving our task in C isn't very different from doing it in Python. I'd even argue that the C version is clearer and easier to follow. This is how it could look like:

A similar thing done in Python obviously requires less scaffolding, but it also introduces its own intricacies via struct module (for unpacking bytes). All in all, it seems like there is not much to be gained here from Python's high level of abstraction.

And that's perfectly OK: no language is a silver bullet. Sometimes we need to do things the quirky way.

After watching the talk for about five times (it hardly gets old), I started to wonder whether it is only those two languages that exhibit similarly confusing behavior... The answer is of course "No", and that should be glaringly obvious to anyone who knows at least a bit of C++ ;) But beating on that horse would be way too easy, so I'd rather try something more ambitious.
Hence I ventured forth to search for "wat" in Python 2.x. The journey wasn't short enough to stop at mere addition operator but nevertheless - and despite me being nowhere near Python expert - I managed to find some candidates rather quickly.

I strove to keep with the original spirit of Gary's talk, so I only included those quirks that can be easily shown in interactive interpreter. The final result consists of three of them, arranged in the order of increasing puzzlement. They are given without explanation or rationale, hopefully to encourage some thought beyond amusement :)

Behold, then, the Wat of Python!

3. Strange loop

None shall pass!

2. Can't catch me

It takes much more to summon Batman, you know.

1. The truth is out there

You had probably seen it coming.... Yeah, you certainly had.

If you know any similar quirks of Python, be sure to present them in comments!

While it might not be immediately obvious what we could use this method for, we can actually find quite a lot of applications if we only pay a little attention. The main advantage of setdefault seems to lie in elimination of ifs; a feat we can feel quite smug about. As an example, consider a function which groups list of key-value pairs (with possible duplicate keys) into a dictionary of lists:

This is something which we could do when parsing query string of an URL, if there weren't a readily available API for that:

So, with setdefault we can get the same job done more succinctly. It would seem that it is a nifty little function, and something we should keep in mind as part of our toolbox. Let's remember it and move on, shall we?...

Actually, no. setdefault is really not a good piece of dict's interface, and the main reason we should remember about it is mostly for its caveats. Indeed, there are quite a few of them, enough to shrink the space of possible applications to rather tiny size. As a result, we should be cautious whenever we see (or write) setdefault in our own code.

Here's why.

This is not the setter you are looking for

A small and obvious, yet important detail: the name of setdefault begins with set. To an untrained eye, it is therefore evident that the method is a kind of setter, and that we can expect a certain behavior to occur if we choose to use it. Specifically, we would assume the method will actually set something, not happily ignore everything we pass to it and do nothing.

But this is exactly what setdefault does, in vast majority of cases! For any pair of arguments k and v, the call:

is almost guaranteed to not do anything. It's only the very first invocation for given k that may actually have any effect - unless, of course, the k is already present in d. Provided we do not remove it, any subsequent call will not really set anything, and this is hardly a behavior we would expect from a setX method.

What should setdefault do, then? Given its name, a more reasonable functionality could revolve around the concept of "default value" for given key, independent from its actual value. It would be returned by [] operator if k were not in the dictionary, instead of KeyError being raised. And any call to setdefault(k, ...) would overwrite the k's default value, whether or not it currently has an actual value. This way, setdefault would deliver on its promise of behaving like a setter.
Of course, this whole notion of "default value" is mighty confusing and doesn't seem useful at all. Even if it was, it could be easily emulated by two dicts working together. In fact, this would be a preferred way of handling such arcane use case; there is simply no room for it in standard library.

Are we really getting it?

As described above, setdefault looks like a deeply flawed version of something which strives to be a setter. But at the same time, it is almost equally eager to pretend to be a getter, for it is often its return value that we are really interested in.

Take another look at the following line in group_assoc_list function:

Notice how we have a setX function which does not only return some value, but definitely a non-trivial one: list! (We are appending to it, after all.) It is unclear where this complex value really came from, unless we are aware that setdefault is actually a variant of dict.get method. In fact, it has exactly the same interface as get - including the return value, which is either an object stored under given key, or a specific default value. Yes, you've heard that correctly: on the surface, setdefault is really the same as get! The only difference is in side effects, which the latter does not produce.

Granted, in group_assoc_list we are vitally interested in those side effects. After all, they build up our dictionary of lists: the very structure this function is about. This might not always be the case, though. Quite often, we will be concerned only with value for particular key, dictionary itself being just incidental.
And this brings us to the next point.

Memoize this!

There is this simple yet powerful - and extremely general - optimization technique, called memoization. The idea behind it is to introduce a layer of "persistent caching" in front of computationally expensive operation. As a result, such operation is performed only once for particular set of input data, with result saved into cache for cheap future access. When we ask for it, the cache is always polled first; if (and only if) result is not found there, it is computed from scratch and then cached (memoized).

What it has to do with dictionaries and setdefault, though?... Well, the common way for implementing memoization is nothing else but a dictionary. Suppose that we have a following expensive function (unary one, for the sake of simplicity):

It looks like an easy task to slap a memoization facility over it. And if we indeed use a dictionary, this also seems like a great application of setdefault, for it does the exact operation we want to perform on our cache: check value for given key, and optionally store one. Overall, it could look a lot like this:

Hopefully we have profiled our code before and after this supposed optimization, because what we are about to find is no performance gain whatsoever! This is easily explicable by the fact that Python is not a lazy language, as it requires all function arguments to be evaluated prior to the call. What it means is that expesive _computation(arg) has to be calculated always, whether or not arg is present in the _cache. It is only afterwards when setdefault performs its check, and that's already too late for it to have any positive impact on performance.

So despite a seemingly perfect interface, setdefault is actually ill-suited for memoization or caching. Changing that would require it to be promoted into full-fledged language mechanism, similar to if-else ternary expressions. This is both extremely unlikely and totally unnecessary, given the narrow scope of resulting construct. As for caching, there are plenty of (correct) alternatives, such as the following:

The last stand

So we have ruled out caching, but what about our original example? While setdefault may indeed suffer from very poor naming, there is little doubt that in group_assoc_list it is the best tool for the job. Or even more generally, setdefault still looks very useful if we want to make sure that specific keys exist in the dictionary - as in this code:

But often it's just a matter of looking at things from different perspective; we can easily stumble on alternative (and arguably better) phrasing:

Finally, even a seemingly ideal solution might be superseded by a more tailored one. Starting from Python 2.5, we have access to the collections.defaultdict utility class - one that readily provides dictionary with defaults. If we use it instead of the ordinary dict, we can pretend like it had all the values already inserted; defaultdict will just create them on the fly.
Here's how our group_assoc_list function benefits from using this class:

Note that defaultdict is still a pretty much dict, so users of above function are unlikely to be surprised by unfamiliar interface.

Final words: be careful

Phew, that was a long writeup, and a summary is in order.

As I've shown, the setdefault method is full of surprises and inconsistencies, causing noticeable confusion. Yet it often creeps up in code written by experienced pythonistas. Sometimes the reasons for it look legitimate, even in face of everything I've said here.
But my statement from the beginning still stands. If you encounter setdefault, raise your alertness - it is might be indirect indicator of code smell. If you find yourself using setdefault, consider thrice whether it's really the best solution. And be especially mindful of the erroneous caching pattern that I gave a lengthy description of.

Even though we do not really know yet how those issues are going to be resolved, there are already few important lessons to be learned here. And I think that the biggest one is not actually concerned with the merit of laws in question. Instead, it draws attention to the problem of communication between IT industry and general public.

It's no longer OK to be ignorant of the Internet

It never was, really. If anything, recent events served as a good wake-up call, reminding that the Internet and all related technological infrastructure is something that we take for granted way too often. While I might exaggerate a bit, I don't think it's very far-fetched to say that for average person, the Internet is pretty much magic. You pop up "the Internet app", type whatever you are looking for, and few moments later, voilà: you just got it, by the sheer magic of intertubes. Assuming, of course, that there is actually something specific you have in mind; otherwise, you can always look at what your friends have "shared", and from there start your clicky journey through the nether. It's awesome, virtually boundless, and it just works... right?

So far, consequences of such technical ignorance were also mostly technical, surfacing as security issues, loss of data, malware spread and so on. But that's alright! We're dabbling into arcane and invoking supernatural powers, so it's no wonder we sometimes accidentally summon few annoying daemons. Should that happen, we can always call an exorcist in a form of friendly geek-next-door, or (at worst) tech support.

Now, however, failure to grasp the fundamental nature of the Internet can result in much more dramatic backlash. See, this technical stuff underneath is not just a plumbing that can be safely ignored. The foundational, idealistic principles of the 'net - decentralization, freedom, knowledge, progress, innovation, flexibility, and so on - are woven into its very fabric. It is by exposure to those "boring details" that those ideas may influence and reshape minds, helping to do away with flawed and outdated notions - including, for example, the 19th century's concept of intellectual property. I cannot even fathom how someone with reasonable understanding of how software, Internet and IT work could conceive something as outrageous as those infamous IP laws. It just doesn't compute.

It is no longer OK to be ignorant of the society

Yet it was conceived, put into words, formed into a legal document and officially proposed - more than once, in fact. Obviously, such things don't happen by itself, especially when powerful interest groups are at play. And that's precisely the reason why we have various public institutions, from parliaments to international organizations: to weed out blatantly bad solutions, and sometimes even let the good ones pass through.

At least, that's the theory. It's naive to postulate virtues among politicians (i.e. those willingly aspiring for power), so we make them cling to one value they'll always embrace, for it is needed to maintain a ruling position: popularity. This should roughly translate to caring for the same things the voters care for. Roughly, because there will be always some disconnection due to effects of scale, perception, biases and multitudes of other factors.

Still, this is pretty much how it works. In order to push an agenda we are vitally interested in (or obstruct one we are strongly against), we need to gain enough publicity. We need to make people share our values and consider as utility those things that we consider as such. That's how we set the appropriate casual chain in motion, eventually leading to fulfillment of our goals.

And this is where the IT industry has failed miserably. It's the reason why we're frantically looking for support, rolling out our biggest cannons, hitting the news with blackouts of absolutely crucial Internet services and DDoS attacks on government sites. We have failed in making the society share our values in elegant, gradual and systematic manner, so now we need to condone a shock therapy to compensate for this negligence. Actually, in some cases we have been actively making things worse, professing the exact opposites of those values mentioned before: centralization, limitation, monocultures, stagnation and rehashing of old concepts.

Now we are just reaping what we have sown.

Those managers - basically a with-enabled, helper objects - are strikingly similar to small local objects involved in the RAII technique from C++. The acronym expands to Resource Acquisition Is Initialization, further emphasizing the resource management part of this pattern. But let us not be constrained by that notion: the usage space of with is much wider.

Interlude on basics

To create our own context manager, we can implement a class with __enter__ and __exit__ methods. The former is called when with block is about to be entered, which is typically associated with obtaining an external resource: file handle, database connection, native memory pool, and so on.

Similarly, once we are about to leave the with block, the __exit__ method will be invoked. If it was an exception that has kicked us out, relevant information will be passed to __exit__ as its arguments; we'll be able to inspect them if necessary.

More details on both of these methods can be of course found in the documentation, which I encourage to peek at.

Universal cleanup

It is perfectly valid for either of those methods to be a no-op, although there is seldom a point in having only __enter__, Context manager with just __exit__, on the other hand, can be quite useful for executing some finishing action that absolutely must be done. If we also have to account for possible errors, custom with is good alternative to doubly nested try statements that would have to be used instead.

Let's look at a practical example. On App Engine, there is a Blobstore service which allows to upload & store big files that are infeasible to keep as text or binary data in database. When it comes to uploading them, most of the relevant logic is already present, so we only have to implement the BlobstoreUploadHandler:

The required part here is performing the HTTP redirect at the end, regardless of whether we successfully did something with the uploaded blob (file) or not. But if "doing something" fails, we would like to delete the blob; otherwise, it will just needlessly take space.

Such behavior can be rather easily contained within a specialized context manager with just the __exit__ function:

If we now put the upload handler's code inside a with statement with above guard, we will always have a redirect issued, and our blobs will be cleaned up properly in case of error:

The great escape (from with block)

Typical uses of with take advantage of its as part, which is almost always used to capture some kind of resource handle. A file object is a canonical example of such practice:

If we want to take advantage of this behavior for our own context managers, we shall return a result from the __enter__ method. It will then be given a name (specified after as), and we'll be able to use it within the with block, as shown above.

But referencing a resource is by no means the only legitimate use for name declared via as. It can actually point to any object, no matter what it is or does.
To illustrate this point, I have came up with rather devious semantical construct which I dubbed a "section". In some way it's the closest thing to goto that we can have in Python without resorting to dirty bytecode hacks. It doesn't support actual jumps (i.e. moving instruction pointer into arbitrary place), but allows to prematurely "bail" out of the with scope, if desired. Hence, it is analogous to the break label mechanism from Java, while also serving similar purpose to infamous do-while(false) construct:

Of course, there aren't many uses for such a thing, and I would prefer not to maintain code that actually employs this trick. But it's entirely sufficient as an example - albeit somewhat silly. Here's how it could be implemented:

As you can see, we're using a special exception as means for "bailing out" of the section. The interesting bits happen mostly inside __exit__, where we examine the type of exception (if any) and the section it came from (to account for possible nesting). We are then returning a boolean value, indicating whether the exception should be squelched.
The __enter__ function is also notable, since it's quite common to just return self from it. This way, the context manager itself becomes the as-object. Not accidentally, this is also the case with the file type, which is what allows for open function to be used both normally and in with-as clause. For section, such behavior is not required, but __enter__ returning self is simply the most straightforward solution.

More ideas

Looking at slightly bigger picture, we can see that with statements create something which might be called a programmable scope. Thanks to __enter__ and __exit__ methods, we are able to capture the events of going in and out of these blocks, and act upon them.

I suspect that when used cleverly, this may result in some pretty powerful and impressive programming techniques, not really related to any sort of resource management. I tried to show a glimpse, but I believe there are many more fruits to be picked from that tree - including:

• Intrinsic profiling. For one, it wouldn't be hard to implement something like a frame rate counter for logic contained in a with block. This could be further expanded into measurements of varying granularity, thanks to with blocks' nesting.
• Parsing. Done by hand, parsing involves rather cumbersome state management. We have to save current position within parsed text, and restore it should we need to try a different variant. This looks very much like a pattern to be encapsulated in specialized with blocks. If done correctly, it could result in nice and neat API that wouldn't hide the control flow of parsing process - unlike most (all?) libraries that rely on explicit specification of grammar.
• Testing utilities. As one almost obvious example, with could be used to check whether test code throws an expected exception. It would be much more convenient (and readable) than assertRaises method from standard unittest package.
• "Important" assignments. As purely syntactic device, we could use with to put emphasis on certain notable assignments in our code. This is vaguely similar to let-in or where clauses from functional languages, but of course it doesn't have any purity implications. The downside: we would often need to use a "fake" context manager if object being assigned isn't already one:
  class var(object):
      def __init__(self, obj): self.obj = obj
      def __enter__(self): return self.obj
      def __exit__(self, et, ev, t): pass

This list is of course nowhere near complete. I encourage to look at the with statement in similarly unorthodox way, for there are likely many semantic gems (or at least polished rocks ;]) to be uncovered here.

But it's usually not too long before our list grows big and it becomes unfeasible to render it all on the server. Or maybe, albeit less likely, we want it to be updated in real-time, without having to reload the whole page. Either case requires to incorporate some JavaScript code, talking to the server and obtaining next portion of data whenever it's needed.

Obviously, that data has to be rendered as well, and there is one option of doing it on the server side, serving actual HTML directly to JS. An arguably better solution is to respond with JSON or similar representation of our items. This is conceptually simpler, feels less messy and is potentially reusable as a part of website's external API. There is just one drawback: it forces rendering to be done also in JavaScript.

While it isn't a hindrance by itself (there are many great tools to help, including Mustache or - sadly discontinued - jQuery Templates), it raises an issue of code duplication. Ideally, we would still want some initial data to be displayed just when the page itself is loaded. And at first, this seems as to require both server-side and client-side rendering functions... I hope I do not really have to explain why it would be a mighty bad idea to create, use and (try to) maintain them.

Client is king

That's why we would opt for rendering exclusively in the browser. Fortunately, it is not only possible, but also quite easy if we already have the logic for handling AJAX calls. Using it, we can prepare initial items in JSON format and put them directly inside some <script> block:

Here, the |json filter represents the before mentioned logic, while renderItems draws the objects-turned items as children of given element. What it does inside can vary from manipulating HTML strings directly (eww!), creating DOM objects (slightly better) or invoking some JS templating engine (best).

Striving for modularity

A simple solution, outlined above, looks sufficient for many practical applications. It is far from ideal, though. For one, it suffers from an acute disconnection, since the actual place where the data is displayed ($('#items')) can be arbitrarily far from the <script> tag. But more importantly, it scales poorly and lacks modularity: should we need to use our list on more than one site, we may need to carefully tailor several moving parts - such as the '#items' selector.

Incidentally, both of those issues can be resolved at once. Let's create a self-replacing <script>: quite similar to the one above, but with some interesting twists. Rather than referencing some DOM element via hard-coded selector, it shall render the items "where they stand": exactly where the very <script> is located. And just as a final touch, the script will even remove itself from the DOM tree when it's no longer needed.

The solution...

Sounds like a pretty clever trick, but it's not exactly complicated at all. In its entirety, it can look somewhat like this:

What makes it work is the fact that var $script = ...; line lies outside of the $document.ready function. Thanks to that, it is executed as soon as the <script> block is parsed - before the rest of document is processed, and way before the $(document).ready callback is invoked.
At that early time, this script is therefore the last <script> element in the DOM tree. In other words, it's exactly what $('script').last() will return! What we are doing in the first line is therefore nothing else, but obtaining a reference to the enclosing <script>.

How's that useful, though?... Well, if we have indeed dropped this <script> right where we wanted our dynamic list to display, its parent() would be the element to render our items in. And because we don't mind cleaning up after ourselves when done, we finish this by remove()-ing the script. Nice and tidy.

...and how to use it

What makes it a pretty powerful technique is easy and flexible integration with modern templating engines. We can almost effortlessly made this snippet into a reusable template part, such as Jinja macro:

The final usage would be then extremely clean and simple:

So why we still cling to somewhat low-level Threads for actual execution, while having no problems with specialized solutions for concurrent data exchange and synchronization?... Well, we can be simply unaware that "getting code to execute in parallel" is also something that can benefit from safety and clarity of more targeted approach. In Java, one such approach is oh-so-object-orientedly called executors.

As we might expect, an Executor is something that executes, i.e. runs, code. Pieces of those code are given it in a form of Runnables, just like it would happen for regular Threads:

Executor itself is an abstract class, so it could be used without any knowledge about queuing policy, scheduling algorithms and any other details of the way it conducts execution of tasks. While this seems feasible in some real cases - such as servicing incoming network requests - executors are useful mainly because they are quite diverse in kind. Their complex and powerful variants are also relatively easy to use.

Let's play in pool

Simple functions for creating different types of executors are contained within the auxiliary Executors class. Behind the scenes, most of them have a thread pool which they pull threads from when they are needed to process tasks. This pool may be of fixed or variable size, and can reuse a thread for more than one task,

Depending on how much load we expect and how many threads can we afford to create, the choice is usually between newCachedThreadPool and newFixedThreadPool. There is also peculiar (but useful) newSingleThreadExecutor, as well as time-based newScheduledThreadPool and newSingleThreadScheduledExecutor, allowing to specify delay for our Runnables by passing them to schedule method instead of execute.

Swapping them

There is one case where the abstract nature of base Executor class comes handy: testing and performance tuning. A certain types of executors can serve as good approximation of some common concurrency scenarios.

Suppose that we are normally handling our tasks using a pool with fixed number of threads, but we are not sure whether it's actually the most optimal number. If our tasks appear to be mostly I/O-bound, it could be good idea to increase the thread count, seeing that threads waiting for I/O operations simply lay dormant for most of the time.
To see if our assumptions have grounds, and how big the increase can be, we can temporarily switch to cached thread pool. By experimenting with different levels of throughput and observing the average execution time along with numbers of threads used by application, we can get a sense of optimal number of threads for our fixed pool.
Similarly, we can adjust and possibly decrease this number for tasks that appear to be mostly CPU-bound.

Finally, it might be also sensible to use the single-threaded executor as a sort of "sanity check" for our complicated, parallel program. What we are checking this way is both correctness and performance, in rather simple and straightforward way.
For starters, our program should still compute correct results. Failing to do so serves as indication that seemingly correct behavior in multi-threaded setting may actually be an accidental side effect of unspotted hazards. In other words, threads might "align just right" if there is more than one running, and this would hide some insidious race conditions which we failed to account for.

As for performance, we should expect the single-thread code to run for longer time than its multi-thread variant. This is somewhat obvious observation that we might carelessly take for granted and thus never verify explicitly - and that's a mistake. Indeed, it's not unheard of to have parallelized algorithms which are actually slower than their serial counterparts. Throwing some threads is not a magic bullet, unfortunately: concurrency is still hard.

This simplicity may be a cause of misconception that the whole problem of letter case is similarly trivial. Actually, I would not be surprised if the notion of having any sort of real 'problem' here is baffling to some. After all, every self-respecting language has those toLowerCase/toUpperCase functions built-in, right?...

Sure it has. But even assuming they work correctly, they are usually the only case-related transformations available out of the box. As it turns out, it's hardly uncommon to need something way more sophisticated.

How Unicode helps (sort of)

Let's do a quick check first. Fire up a REPL for your favorite programming language (or editor with a compiler nearby) and try to uppercase all characters in any of the following strings:

• Falsches Üben von Xylophonmusik quält jeden größeren Zwerg.
• Voix ambiguë d'un cœur qui au zéphyr préfère les jattes de kiwi.
• Pchnąć w tę łódź jeża lub ośm skrzyń fig.
• Экс-граф? Плюш изъят. Бьём чуждый цен хвощ!

These are the pangrams: short and often nonsensical sentences which contain all letters from their respective languages. Since many of those letters are outside of the a-z range, their uppercase counterparts cannot be obtained by simply subtracting a magic number 32 from their ASCII code. Because they are scattered over much wider span of character table, this simple approach doesn't work.

But programming languages usually promise to handle all the complexities behind upper- and lowercasing - if not for strings, then at least for individual characters. Many of them, however, fail to deliver on that promise in their standard strings API. It's the Unicode, of course, that is supposed to deal with such issues. Using Unicode strings by default is rare among popular languages, though; Python 3.x, Java and Haskell come into mind as honorable exceptions. Others typically offer alternate string types (such as std::wstring in C++ or unicode in Python 2.x) and various encoding & decoding mechanisms to cooperate with code that uses standard, ANSI-only strings.

Case in point

Still, we may consider this an appropriate abstraction when it comes to handling letter case of individual characters. However, it isn't nearly enough for actual texts. The issue of proper capitalization is grammatical one; and whenever grammar of natural languages is involved, we can be sure that any related computing problems will be notoriously difficult to solve.

I stumbled upon this very topic when making the switch to blogging in English. It turns out that English has quite amusing rules regarding capitalization of titles - or title case for short. Its simplest variant is to capitalize every single word:

Conventions which are actually used are much more complex: they introduce several exceptions, and exceptions to exceptions, so it all contributes to rather fine mess. For example, it is common to leave certain "small" words lowercased: prepositions (for, by, on, ...), articles (a, an, the) and conjunctions (and, or, but, etc.). Nevertheless, the first word is always capitalized - and depending on particular convention, the last one may be too. Additionally, if the "small" word grows big enough, it might also warrant capitalizing; an excessively long preposition such as without could be a good example. Finally, punctuation may also play its role, forcing any word after certain marks (such as colon) to be capitalized as well.

That's rather convoluted, isn't it? Setting aside any concerns about how arbitrary these rules are, we can observe that even an undisputed standard built upon them is unlikely to be easy to implement as an algorithm. Anything that requires recognizing parts of speech is almost certain to be AI-hard. Although a simple exclusion list (['a', 'an', 'but', ...]) works correctly for many inputs, it fails for texts containing e.g. certain phrasal verbs. So in order to properly capitalize a title, one should know a semantic context of every word in it. That's easy for humans, but not so for computers.

In any case...

I hope I have successfully made a point that changing letter case may be almost as far from trivial as a computing problem can get. At the very least, we should be extremely wary of any related operations on texts in foreign languages - including, for example, a case-insensitive comparison. And even for English texts, the task may border on impossible if it's something just slightly more complicated than straightforward substitution of characters.

For more information about the phenomenon of letter case, it's probably best to consult the indispensable Wikipedia.