It's all about this map (click for interactive version)

To my datavis readers, sorry for that string of posts in French but what better data to visualize than political data, and what better time to visualize political data than election time, and what better audience for such visualizations than the folks who are asked to vote?

Like last time, though, I am writing a follow-up technical post about how I dealt with the issues of this visualization.

So anyone who ever tried to make data visualizations knows that you can hardly start without data.

My ingredients for the recipe were:

2012 presidential election results by circonscription, plus those of 2007.

Results of the previous congressional election. There were 2 files, one per round, as opposed to a flat file of députés in place (I didn’t find one that didn’t required some significant editing to be of use). Most importantly, I needed their political orientation which required some tweaking.

Matching tables between circonscriptions and cities.  From a previous project, presidential election data at the city level. Also, geo coordinates of the cities.

The data which was most painful to extract was the list of candidates. In all fairness, UMP made it easier than PS as they had them all on a page. For PS, they had a google fusion table which had this as a data source. That file required a lot of massaging. Eventually local pages of the PS site would list the candidates missing from the map (or provide alternate names). When it was up, I also used the http://www.elections-legislatives.fr/ site to check for the missing names.

Finally, I figured out the genders of all the candidates by extracting their first name and looking up all the ones I wasn’t sure about (there are quite a few unisex first names in French).

Now calculations.

There is a pretty strong statistical link between the score of a party on an election in a certain territory, and the chances of a congress candidate of the same party of winning the district.

Predicting these chances is a well-known problem known as classification  for which the textbook method is logistic regression.

All we need was the 575 districts for which I had results. We then associate the score of a party at the 2nd round of the election to whether the corresponding congress candidate got elected (1 or 0). That gives us 1150 pairs of values which we throw in the mathematical cooking pot.

And what we get is the following formula:

where x is the score in the previous election (between 0 and 1).  As you can see when x gets close to 0, the denominator becomes a very large number and the probability quickly drops to virtually nothing, and converserly when x gets close to 1, the denominator becomes very close to 1 so the probability rises up to 1 equally fast.

With this and that in place, it is possible to come up with a reasonable estimation of the chances of any candidate based on the recent results. As an aside, the current Prime Minister has renewed the tradition started by his predecessor to ask ministers to seek office and to force them to step down if they fail to win their district. As a result, 24 out of the 37 ministers are campaigning. Out of those 24, 2 are taking very serious risks according to this model: Marie-Anne Carlotti and Benoît Hamon.

Finally, geography.

In an ideal world, there will be an abundance of geoJSON files describing France and its many administrative entities. Usable data must exist somewhere, because the maps on www.elections-legislatives.fr have all been generated (by Raphael.js says the source code). If I’m doing another project on these elections I might reverse engineer the shape of the maps to extract the coordinates.

Without a dataset, the work of drawing the boundaries of 577 districts is just huge. However, accuracy is not required as I’m only putting the districts on a map so people can look up where they live or places they know. In my previous work in order to let users change the composition of the districts, I wanted to be rigorous in the placement of everything but here we can live with imperfection.

So I am using the same principle as I did: voronoi tesselation.

For each district I am picking the largest city, for which I have the coordinates. But most large cities belong to several districts. So I am adding random noise to each point. Then, I am drawing shapes around them.

That would normally fill a rectangle, so in order to make it look like France, I have drawn a clipping mask on top of it (that, I’ve done by hand, picking coordinates of the outline of France).

That about wraps it up!

Cliquez sur l'image pour arriver à l'application interactive

J’ai eu l’idée de créer cette carte quand j’ai pu mettre la main sur les données des résultats de la présidentielle par circonscription législative. En 2007, les résulats du deuxième tour ont été un très bon prédicteur de ceux des législatives qui ont suivi: une circonscription pour qui un candidat à la présidentielle au second tour a fait ne serait-ce que 52% a plus de 75% de chances d’être remportée par le député du même parti.
Et encore, tout le monde s’était accordé à dire qu’en 2007, la gauche avait fait une très bonne campagne et qu’elle avait endigué la vague bleue. Les probabilités sont sûrement encore plus élevées.

Or, presque 80% des circonscriptions ont été gagnées avec un score supérieur à 52%, ce qui est donc une très grosse marge. Dans plus du quart d’entre elles, le gagnant du scrutin a même récolté plus de 60% des voix…

Bref. Dans la plupart des circonscriptions, il n’y aura pas tellement de suspsens. Là où je vote, on n’a pas souvent le droit à un deuxième tour.

Comme pour le découpage électoral, je trouve ça un peu dérangeant. L’élection n’est pas tant la rencontre entre une personne et une population qui la choisit, mais surtout le fait d’un parti qui place ses pions, surtout si on rajoute les “accords électoraux”. Je crois que j’ai moins de chance de rencontrer la candidate challenger de ma circonscription que de voir François Hollande ou Nicolas Sarkozy “en vrai”.

Alors pourquoi ne pas en profiter pour s’approcher de la parité à l’assemblée?

Pourquoi ne pas le faire d’abord: si un parti ne présente pas autant de femmes que d’hommes, il récupère une amende. Ou plutôt, comme l’explique bien Alexandre Léchenet,  il perd des financements. Mais le mode de calcul est biaisé. Un parti récupère une certaine somme par voix  au niveau national, puis cette somme est minorée si les femmes représentent moins de la moitié des candidats. Pour contrer ce système, il aurait été plus judicieux de le baser sur la proportion des voix remportées par des femmes, pas sur celles qu’on a alignées au départ.

Donc, on envoie les femmes au casse-pipe: on les met dans des circonscriptions impossibles à gagner, histoire d’éviter l’amende. Il y a presque 100 candidates qui se retrouvent contre le représentant d’un parti qui a fait plus de 55% au second tour.

A Paris, par exemple, Annie Novelli défie Claude Goasguen dans la 14ème circonscription (qui a voté à 77% pour Sarkozy) et c’est Agnès Pannier qui affronte Bernard Debré dans la 4ème (75% pour Sarkozy). Pendant ce temps, Roxane Decorte se frotte à Daniel Vaillant dans la 17ème circonscription (qui a voté Hollande à 72%).

Et encore, en théorie elles pourraient gagner, mais dans 238 circonscriptions, soit plus de 40%, ni le PS ni l’UMP n’ont investi de femme, comme ça on est tranquille.

Ce qui fait qu’au final, même s’il y a 40% de candidates, le nombre d’élues devrait tourner autour de 175 soit 28%. Ce serait quand même presque 70 de plus qu’aujourd’hui, malgré le cynisme de l’actuel patron de l’UMP. Espérons qu’elle pourront aller à l’Assemblée habillées comme elles veulent.

Mine de rien, il y aurait 51.5% de femmes en France.

First, getting the data. I already scraped the results of both rounds of the presidential election by city. The districts for the congress election are also known, but it’s not possible to do a match, because large cities are almost systematically broken down into several such districts. Paris, for instance, will be represented by no less than 18 députés.

So I needed the results by the finest possible unit, that is by individual polling station. During the election night these results are compiled by city and centralized, so you would assume that the raw data of each polling station is available somewhere. That is not the case, unfortunately. Although it seems that they will be made public eventually, that may not be the case before the June 2012 election.

Fortunately, Open Data Paris had the results by polling station. More: it had their address and matching of every inhabited building in Paris to its corresponding polling station.

To map the polling stations, my first intuition was to create a voronoi tesselation of their projected, geocoded coordinates (I only had their addresses in the raw data file). In short, voronoi polygons can be generated for a certain number of control points and correspond to the area nearer to that control point than to any other. So it’s a good approximation of the areas  which correspond to a given polling station.

Problem: several polling stations could be in the same address, and for the voronoi algorithm the control points have to be distinct. So I tried jittering them (adding random noise to each one). A tesselation could be done that kind of looked like Paris but voting districts will look messy as there were frequent inversions between neighboring districts.

So I had to come up with a better approximation of what part of the city corresponded to what voting district. So I used the address to polling station correspondance, and for each polling station I took the first and the last street number of any street that was covered by it. Then I geocoded the whole lot. That’s about 16000 points. It took some time.

Here's my polling station as an example.

Then, for each polling station area, I took the minimum and maximum longitude and latitude, which formed a bounding box, and assigned the polling station to the center of that box. Then, I used tesselation again.

I found a number of oddities in the geocoding that I had to correct manually, because if one address was not accurately coded, chances are it would change the shape of the bounding box drastically and so the position of its corresponding polling station. Sometimes the geocoding service wouldn’t find the street and/or would use a street of the same name in another city, sometimes they did find the street but the coordinates were way off… So the dataset required a lot of massaging before it got into shape.

The last geographic errand I had to do for this visualization was to create a perimeter of Paris to use as a clipping mask, else the tesselation would be done on a rectangular shape with the edge polygons being very large and very skewed. So I collected coordinates of points around Paris to create one polygon. Only what’s inside of this polygon is shown (.style(“clip-path”) in d3).

After the data has been acquired, the building of the rest of the datavis was nothing special. I have used extensively mouseover and click events to trigger transformations as I always do, although this time I did prepare a lot of rules.

Originally I wanted to make the whole of France like this, though it will be difficult: one, to get the data, and two, to get it into shape. As of today the location (i.e. street address) of most of the polling stations is not available online, so even if we got the number of votes for each of the polling stations (there should be about 40000 of them) the geographic part of the problem will remain unsolved. Though, it’s a worthy endeavour. While the election results have little interest at a macro-geographic level – by region or by département – they are very useful at a very fine level as strategies can be constructed.

For instance, it’s worthwhile to send heavyweights to conquer districts that are winnable, but it’s a waste to keep them in their respective fiefdoms if victory in these districts is already certain. Also, when districts would have to be redefined, having this kind of information can be invaluable to the political force which gets to draw their new limits, or to their opponents.

Et surtout il permet de changer la composition de ces circonscriptions, dont le tracé aujourd’hui est assez arbitraire. Il y aura 18 circonscriptions à Paris contre 21 aujourd’hui, et elles ne suivent pas les arrondissements.

Le tracé de ces circonscriptions est déterminant pour le résultat des élections. Aujourd’hui, par exemple, il y a deux circonscriptions où Nicolas Sarkozy a récupéré plus de 75% des voix au 2ème tour de l’élection présidentielle, j’imagine que la gauche ne place pas trop d’espoir sur leur reconquête. De même, il existe pas moins de 9 circonscriptions où François Hollande a reçu plus de 60% des votes. En l’état actuel des choses, 12 circonscriptions semblent acquises à la gauche et 6 à la droite, dont 3 pourraient peut-être quand même être gagnées par la gauche.

Le découpage actuel n’est optimum ni pour la gauche, ni pour la droite. En modifiant le tracé des circonscriptions, la gauche pourrait toutes les remporter, et la droite pourrait en gagner 12 sur 18 (ou peut-être plus, 12 restant mon high-score personnel). Pour favoriser un camp, l’idée consiste à répartir les bureaux de votes les plus favorables entre le plus de circonscriptions possibles, plutôt que de les garder dans peu de circonscriptions. En généralisant sur le territoire, on imagine ce que ça peut donner!

Donc, quelque soit le sentiment actuel, tel ou tel redécoupage peut complètement redistribuer les cartes. C’est un sentiment dérangeant parce que ces redécoupages arrivent régulièrement et sont relativement opaques. D’ailleurs, il est assez difficile de faire le lien entre les données des élections présidentielles et les circonscriptions législatives parce que les résultats ne sont que rarement disponibles par bureau de vote.

Je donnerai les détails techniques de l’implémentation dans un futur post.

Qui dit élection présidentielle, dit résultats.

Et qui dit résultats dit représentation visuelle dans les médias.

Et donc souvent carte.

Cela dit, une carte ne nous apprend pas grand chose de ce qui s’est vraiment passé dans une élection. C’est vrai que c’est utile pour retrouver sa région ou sa ville, mais les résultats de deux villes proches géographiquement (par exemple, Boulogne-Billancourt et Issy-les-Moulineaux) n’a strictement rien à voir.

En revanche, avec une carte c’est très difficile de répondre à certaines questions, comme: où a-t-on le plus voté pour Nicolas Sarkozy? Qu’est ce qui s’est passé dans les villes qui ont beaucoup voté pour Marine Le Pen ou François Bayrou? D’où vient l’explosion du vote blanc au deuxième tour?

Pour répondre à ces questions, on peut utiliser les coordonnées parallèles.

Chaque axe vertical correspond à un vote possible – d’abord ceux du premier tour, puis ceux du deuxième. Chaque ligne de couleur correspond à une ville (ou à un département, à une région, ou à la France). Chacune de ces lignes coupe chaque axe à une hauteur qui correspond à la proportion des gens qui ont fait tel ou tel choix. Par exemple, les villes où on a beaucoup voté pour Jean-Luc Mélenchon, comme Bagnolet ou Gennevilliers, se retrouveront vers le haut de l’axe du milieu.

En s’approchant des axes, le curseur devient une croix. Il suffit alors de le faire glisser le long de l’axe pour dessiner un rectangle, et mettre en valeur toutes les lignes qui passent par ce rectangle. Par exemple, voici les villes qui ont donné plus de 70% de leurs voix du second tour à François Hollande:

On peut tracer autant de rectangles qu’on veut. Par exemple, on peut ne garder que les endroits qui avaient soutenu Bayrou (par exemple à plus de 10%).

et voilà: il n’y en a plus que 2.

Pour supprimer une sélection, il suffit de cliquer près de l’axe sans glisser. Si on clique sur un rectangle, on peut aussi le faire coulisser le long de l’axe.

Avec cette technique, on peut tout de suite voir les fiefs de tel ou tel candidat: il s’agit des lignes qui touchent le haut du graphique.

On peut répondre à des questions plus complexes. Par exemple, je parlais de l’explosion du vote blanc au deuxième tour. Que s’est-il passé?

Sélectionnons les endroits où le vote blanc a dépassé les 4% au deuxième tour, tout en restant à moins de 1.5% au premier tour:

On voit trois pics: les villes où Marine Le Pen, Jean-Luc Mélenchon ou François Bayrou ont fait un très gros score. On peut continuer à sélectionner et voir que ce sont bien les villes qui sont dans la pointe de l’un des trois triangles et pas celles qui passent par le bas qui se retrouvent dans ce cas de figure. Ces électeurs ont préféré voter blanc plutôt que de choisir.

Bonne exploration!

http://www.jeromecukier.net/projects/elections/dtour2012.html

Three things to know about the conference. It’s one of the main events on visualization in Europe. The main event takes place in a church which sits hundreds.

Also, the conference is not called die Konferenz zur Visualisierung von Information for no reason. It’s mostly German-centric.

Finally, the conference’s approach to talking about visualization is to offer some height of view by inviting experts who do not work in information visualization proper but offer an interesting perspective to the field from their point of view.

Videos from all talks will be made available shortly.

The first speaker was Dr Thomas Henningsen from Greenpeace and was about how the organization is using visual impact for their agenda. Greenpeace goes indeed to considerable lengths to take the one picture that shows that what they are fighting is not an abstract possibility but something very tangible. Sometimes they create the picture as in this example,

sometimes they capture a certain moment, but they also use charts and data to make their point.

Next was Prof. Dr. Norbert Bolz. From the opinion of many of the native German speakers I heard this was the highlight of the day. As my German is somewhat rusty and as he didn’t use visual aides I admit I missed a lot of it. Prof. Bolz is presented on the see conference site as a media scholar and many of his books were on display at the conference. His talk was on how can one inform and be memorable through images. So here are parts I grabbed. To be remarkable a piece of information has to be new, it has to be on something that the reader didn’t know. This is very different from being important. A lot of the talk was also about the memorable faculty of images (Prägnanz), that images can’t be cancelled.

The next speaker was Stefanie Posavec who describes herself as a data illustrator. Like most people, you may think of data visualization  as a supervised but automated process by which a computer generates an image based on data and a set of rules. Then, enters the data illustrator, who works entirely by hand.

Perhaps Stefanie’s best-known work is the representation she’s done of Kerouac’s On the Road:

which most people assume is a fine example of generative art. Wrong: every. single. element. is. placed. by. hand.

Stefanie took us through her project and shared how she works, how she collects data and encodes it (she did bring a computer).

Here’s a picture I took at the workshop the next day along with some of her sketches.

Stefanie always seems to be apologetic that she does not write code which is ironic considering that this approach is what makes her work unique. During the workshop, she took us through specifications that she had written for a developer to create an interactive visualization and which were insanely detailed. Everybody who writes code would be really thrilled to be able to rely on such a structured document!

The next speaker was Ben Kreukniet from UnitedVisualArtists. Now UVA may not be a household name, but how about Red Hot Chili Peppers or U2? Remember how everyone was talking about the gigantic scenic structure on the U2 last tour, which AFAIK was the highest-grossing concert tour ever? that was UVA’s work. They are lighting artists who specialize in large installations, and by large they mean friggin’ epic.

Ben’s talk took us through their work, with a focus on their “origin” project:

Origin is a gigantic cube of light and sound which is made to interact with an audience. During the workshop the next day, UVA showed us the tools they use to work, which revolves around a platform they call d3 (though not that d3). And we had the privilege to preview their next work which would be an advertisement campaign to be shown in movie theaters.

The next speaker was Yannick Jacquet from the antiVJ collective. More than a portfolio talk, this was an introduction to what VJing is about for the many of us who only had a vague idea. Basically, VJing is about showing moving pictures. But it doesn’t have to be ugly psychedelic shapes moving on flat screens in night clubs. antiVJ was created in reaction to this reductive view of the field. Through a technique called video-mapping, VJs can use one projector to cast images on many separate surfaces, and with a couple of projectors which can be controlled from just one computer, they can cover very complex geometries.

Next was Michael Madsen. I covered his workshop talk in my previous post, but during the conference he presented his movie “Into Eternity”:

Here’s the story. In Finland, law requires that nuclear waste be disposed within the country. So a company is building a bunker to bury it deep, deep within the earth – eventually, canisters of nuclear waste will be stored 4000 meters underground. In 2100, the facility will be reaching its capacity and it will be sealed and expected to remain undisturbed for the next 100 000 years. This is the first human creation designed with such a horizon. Contrary to religious buildings which are being built “forever” everything has been done to give the facility the highest chances to last 100 000 years. This led to surprising choices. First, the location of the facility is secret. When it will be sealed, there will be no distinctive mark at all. And while there is a possibility of a next ice age or a similar global disaster and with that the possibility that the security be breached. This is precisely on what the movie is about – which decisions were made, why, with what perspective.

So documentaries are all about telling stories visually, which could also be told of data visualization. Only for documentaries, the angle – a combination of the subject, the approach, the questions that need to be answered – is the result of an overly elaborate research, which we sometimes do in datavis, and sometimes don’t. More often than not it’s tempting to just go ahead with a form that agrees with the dataset though, so this work process is a welcome perspective.

The last talk of the conference was by none other than Manuel Lima, who pioneered visualization blogs with visualcomplexity. Both his talks (to the exception of the description of his experience at Bing) relate to material in his highly-recommended book, Visual Complexity: Mapping Patterns of Information, the See conference talk focusing more on trees and hierarchical displays of information and the second one more on networks.

Trees can be taken literally

This was definitely the closest talk to actual datavis practice. The principles he exposed really come to life with the examples, so I will encourage you to watch his talk in its entirety (it should be available on the 12th of May). In passing, in his workshop talk he mentioned that most of his “ancient” examples come from an out-of-print grimmoire called The Album of Science, I found one for $4 on Amazon so you may want to check it out.

That about wraps it up.

Pros of see conference:

• really cheap to attend from anywhere in Europe – transportation, accommodation and conference fees are super reasonable.
• big.
• not just about datavis, but rather on subjects which are particularly interesting for the datavis practitioner.
• in pleasant Wiesbaden.
• party-time/conference-time ratio about optimal.
• workshop is really, really fantastic (and free to attend btw).

Cons:

• you’re kind of expected to speak German.

The theme of See+ was tools. Speakers were invited to tell us how they work.

One characteristic of the See conference is that it offers a broader perspective on data visualization. In addition to well-known datavis specialists, such as Manuel Lima this year, other guests include visual artists and designers who also work with data, as well as experts in communication who apply that skill to data.

Michael Madsen is a Danish film-maker who lives in Berlin. In his see conference talk, he presented Into Eternity, a powerful documentary about an incredible facility in Black Mesa Finland which is designed to hold nuclear waste for 100,000 years. While the eerie tone of the narration is fascinating this was perhaps the topic most remote from “traditional” datavis (as in data, graphs and stuff).

Then came his See+ talk which took a turn I didn’t expect.

Pic courtesy of Joshua de Haseth. I'm actually the guy on the right.

Filming a documentary takes at least two years by a conservative count. A filmmaker first worry is therefore to find what will drive them for that long. More than a 9 to 5 job, more than a simple theme, what they are looking for is a vision, one unique way to treat one unique subject. Michael elaborated on the difference between executing what you are told to do, and a fulfilling calling that comes from the self. The flipside is that it is very difficult to go through times of no project. One subject powerful enough to give one a reason to work everyday for years does not simply come by; it can take months or even years of doubt before one is found. In Michael’s words: when I have no project, I have no identity. But when he does, his thrill is to seek and explore as he is doing something that has never been done before.

(I’m leaving a lot out which is more directly related to film-making.)

As he was discussing that I could sense people in the room tune to this, as I did. We datavis practitioners are all makers, tinkerers, inventors. Fortunately, our work cycles are often shorter and it is easier to start a new one. Certainly, there are patterns and recipes and things that need to be done for work. There are also hacks found on stackoverflow (thanks for that) and inspiration from the works of other (and thanks for the debugging console).

But there is also a “great unknown” in visualization – data that has never been collected let alone represented, techniques that have never been used, combinations that have never been tried – and things that can not even be put in words. All of this requires curiosity, independence and dedication. And the outcome may not live up to expectations. But it’s just a reminder that to do what we do we must leave our comfort zones and our ways, set off and explore.

Noah Iliinsky and O’Reilly were kind enough to send me one review copy of Noah’s book and who says review copy says review, so here goes.

We need more introductory books to data visualization.

I’ve had several discussions with data visualization colleagues who feel that there are too many books already. I strongly believe otherwise.

As of this writing, there are 59 books tagged data visualization on Amazon, versus well over a thousand for Java (for example). And on those 59, I would say about a dozen qualify as introductory. Here are 3 reasons why introductory books are important.

• You only need to know a little to start making effective visualizations. A small book won’t teach you all there is to know about visualization, but you don’t need that to get off to a good start. A lot of this has to do with asking yourself the right questions. But this is a very unnatural thing to do, especially when you feel you can do stuff. Fortunately, even a short book can help you to pause and think.
• An effective visualization is not harder to make than a poor one. Well, actually it is, really good visualizations are built after many iterations on one promising concept. But the point is, a lot of efforts and ressources can go into abyssmal visualizations. If you are in a position to buy visualization, having even basic knowledge of how data visualization works can prevent you from wasting your money.
• There are many approaches to visualization. The right introductory book will be the one that resonates with you. Some people who are interested in this love to code, some are afraid of programming. Some are accomplished visual artists, some don’t know to draw. Some have specific needs (business dashboards, presentations, interactive web applications, etc.).

Where does designing data visualizations fit?

Designing Data Visualizations is a very short book – the advantage is that you can read this in a couple of hours. It’s perfect for a train or plane trip for instance. The format of the book (23 x 17.5 cm, flexible paperback) makes it easy to carry and read anywhere. And it’s an easy read – you won’t need to put down the book every few pages to make sure you understood.

The flipside of this is that you won’t learn any actionable skills from the book. The book is never trying to teach you to make things : this is explicitly outside of its scope. What is does is make you think on how to do stuff. It makes you consider the choices you make.

So you’re making a visualization. Does your choice of representation makes sense? how about your colors? placement? If you’re not confident that you know the answer to this kind of questions you must read the book right now; else, you won’t be able to improve your work. And again that is what successful designers do – iterate and improve, again and again and again.

As a non-native speaker of English one reason why I enjoy reading introductory books is for their excellent formulation of things. You know, there are those things you have a vague idea of, and the writer puts the exact words on it. So I’ll go ahead and quote my favorite paragraph :

Consult [your goal] when you are about to be seduced by the siren song of circular layouts, the allure of extra data, the false prophet of “because I can”. These are distractions on your journey. As Bruce Lee would say, “It is like a finger pointing a way to the moon. Don’t concentrate on the finger or you will miss all that heavenly glory”.

Who is this book for?

I think the people who would benefit the most from the books fall in two categories:

1. Those who know absolutely nothing about visualization but have some interest in the subject. And the subset of those who don’t really have time to find out all about it (think: your client, your n+2 boss). They will appreciate that there is a real take-out value in such a short book.
2. Those who can create visualization because for instance they are coders, designers, excel users etc. and who see data visualization as a byproduct of their activity, so they never really asked themselves those questions. And among those, I’m thinking mostly of coders. Noah and I met at last year’s Strata conference which is also attended by the cream of the crop of the data scientists. I was surprised to see that some of them, despite being able to harness huge quantity of data, were severely limited in their visualization options because they never had an opportunity to learn. These people who are already at ease with the tool will see their activity supercharged thanks to the book.

How did we end up with this?

it’s really cool working around movies, because it’s something we can relate to.

A part of my movie ticket stubs stash.

At first I wanted to do something out of keywords we could grab on the movies but  Jen came up with another idea I found more worth pursuing: working around the story types (which was the most interesting aspect of the curated contest dataset) and see if there was not some kind of grand truth we could unravel there. She also requested stars and glitter, because we were not going to work on this glamorous dataset with a tedious dashboard done in Excel.

That truth didn’t take so much time to find: the most frequently used story types (like comedy or movies with monsters) do not perform well in the box office while different story types (stories of teens growing up, or when the main character turns into something else), which are used less often, are much more profitable. So why doesn’t hollywood make more Junos and Black Swans and fewer College Road Trips or Dylan Dogs?

That’s the idea. Now the making.

Fair warning – the rest of this post is fairly technical. 

Making stars

If I had to contribute significantly to the project it had to be done in d3/svg.

Fortunately, it’s easy to generate star shapes in d3. Once you have the coordinates of where the points of one unitary star should be, you can easily make stars of any size with a function and a parameter.

var c1=Math.cos(.2*Math.PI),c2=Math.cos(.4*Math.PI),
    s1=Math.sin(.2*Math.PI),s2=Math.sin(.4*Math.PI),
    r=1,

    // ok the constant after r1 is the thickness of the branches.
    // 1 is a "straight" star, less is narrower, more is thicker.

    r1=1.5*r*c2/c1,
    star=[
        [0,-r],
        [r1*s1,-r1*c1],
        [r*s2,-r*c2],
        [r1*s2,r1*c2],
        [r*s1,r*c1],
        [0,r1],
        [-r*s1,r*c1],
        [-r1*s2,r1*c2],
        [-r*s2,-r*c2],
        [-r1*s1,-r1*c1],
        [0,-r]
        ];
    // this is a list of the pair of coordinates of the points that make a star.
lineStar=function(k) {
	var line=d3.svg.line()
		.x(function(d) {return d[0]*k;})
		.y(function(d) {return d[1]*k;})
	return line(star)+"Z"; // this will stitch everything together.
}

Now, running lineStar(10) will return the path description of a star with a radius of 10, thusly:

"M0,-10L3.367709824346891,-4.635254915624212L9.510565162951535,-3.0901699437494745L5.449068960040206,
1.770509831248423L5.877852522924732,8.090169943749475L0,5.729490168751577L-5.877852522924732,
8.090169943749475L-5.449068960040206,1.770509831248423L-9.510565162951535,-3.0901699437494745
L-3.367709824346891,-4.635254915624212L0,-10Z"

Placing, moving (and spinning) the stars

The next idea was placing the stars.

And for this we need two things: being able to position them somewhere, and being able to move them easily from point A to point B, ideally with some cool effect in between.

So, it would be possible to change the x and y attributes of the path, but each would have to be dealt with separately with a different function call. I found it a better approach to rely on the transform attribute and translate. Each time I want to position a star somewhere, I need it to be set at an x and y coordinate, which will always correspond to either the data of the star, or that of a group above it. For instance, a star corresponding to a movie will need to be at the position corresponding to the data of that movie, or that of the story type above it if it’s still collapsed, or that of the high-level grouping of story types if that’s collapsed.

Now all of the data structures for that are array of objects which all have x and y keys. In other terms, for any star-shaped object, I can always expect the underlying datum d to have d.x and d.y values. So, I wrote a function translate(d) which works on those 2 properties. And as a result, when I need to position any object all I have to write is:

.attr("transform",translate)

and the object will be positioned according to its underlying data. (this is equivalent to writing .attr(“transform”,function(d) {return translate(d);}) )

If I need to be them elsewhere, i.e. at the position of their parent, I can pass the data of that parent as an argument, for instance:

.attr("transform",function(d) {return translate(structs[d.struct]);})

For a cheap bit of extra action, I’ve added a spinning effect in the translate function. Since translate(d) returns a value for the transform attribute, nobody said it just had to be instructions for translation! so I’ve added a rotate after the translate. The arguments for the rotate function depend on the x and y properties of the argument as well, so when stars move across the screen, the rotate angle changes slightly with each increment of either coordinate, giving the impression of spinning.

Explosions, starlets and other effects

Most of the cool things happening in the visualization rely on one very simple principle about d3 transitions: chaining them.
In the code you’ll find oftentimes this pattern:

.selectAll("someobject").data(...).enter().append(...) // creates the items
... // sets the initial attributes
...
.transition()
... // change the attributes
...
...
...
.each("end", function() { // stuff to be done on each item after the transition is over

and within that function, you’ll find either:
another transition which starts exactly when the previous one ends, so for instance opacity can decrease (causing a fading effect): d3.select(this).transition()…

or a command to remove the object: d3.select(this).remove().

When another transition is called, there can be another one after, then another one, then another one, then eventually the object can be removed (or not).

Now you may think of transitions as ways to get one object to change smoothly from state A to state B, like a rectangle moving across the screen. But if you start to think that the objects can be discarded after the transitions, you’ll realize that there is an unbelievable number of things that can be done with them.
For instance, upon clicking on some stars, I am creating another star shape at that same location. Initially it has a the same size as the star, but I increase that radius to a large number (1000px) while decreasing its opacity to 0. So it seems that the new star is both exploding and fading. When it’s become transparent I remove it.

gStructs.append("svg:path") // here I'm creating a "path" shape
.style("stroke","none") // with no outline
.style("fill",colorXp)  // with the fill color of the explosion
.style("opacity",0.2)  // and a low opacity to start with (translucent)
.attr("d",lineStar(d.size[sizeAxis])) // I give it the shape of a star and the size of the
                                      // star that's being clicked
.attr("transform",translate(d)) // and I position it on that star

.transition() // action!

.duration(500)	// a 500ms transition. Long enough to see the effect.
.attr("d",lineStar(1000)) // the star expands to a radius of 1000.
.style("opacity",0) // while fading to transparency.

.each("end",function() {d3.select(this).remove();}) // and when it's done - it's removed.

Changing axes

In this visualization I let the user change what’s plotted along the axes. It’s not very difficult to do but it’s a hassle to do it late in the project as it has been our case because it requires a lot of housekeeping. This is really about the data structures that will support our items. Instead of having just one value for x, y and size they have an object with several keys, one per axis. Then we maintain one variable per axis type, so everywhere we should write: d.x, we write instead: d.x[xAxis].

So when there is an axis change, of course, we do a transition so that the stars and everything move smoothly to their new position. But what if the objects were already moving? When an unplanned transition interferes with an ongoing one, the results are often ugly, especially if the current transition had chained transitions waiting to be triggered. In other words, this will leave a mess.

The way I’ve dealt with this is by keeping a tab on the number of transitions going on at a certain time. The axis change could only occur if no other transitions were taken place. If that was the case they were simply denied. There are other ways to do that like a queue of actions but that seemed the simple and adequate way to deal with this.

Bootstrap and google fonts

This was the first non-trivial project where I used bootstrap and I’m just never going back. Bootstrap simply removes all the hassle of arranging all the elements of a visualization on a screen and is very easy to use. Plus, it comes up with sensible options for buttons, forms, and the like. Since the contest it has evolved faster than a pokémon, for instance it is now possible to specify custom colors in a form and bootstrap will generate the appropriate css files. Google fonts are another great help as they are a very easy solution to choose fonts among a relatively large number of choices without relying on the fact that all the users have these fonts on their computer.

Wrapping it up

There’s a lot of other hacks in the code which you are welcome to explore, I admit I don’t remember them all because I took too much time to write this blog post after creating the entry (bad). However if there is one point you would like be to explain please ask in the comments.
I’m not entirely sure of what happened when I submitted the entry though. First it wasn’t listed with the others, then I got a message saying it hadn’t been reviewed, so it didn’t win anything, yet some time after the prizes have been handled it appeared in the “shortlisted” visualizations for the contest (which I found by accident). So whether or not it was good, I let you guys judge, at any rate it was fun making.

A complex treemap

• we use the polygon mark,
• we generate the treemap layout outside of tableau.

Simpler but dynamic treemaps

Data

In Tableau

case [rectangle]
when "no vote" then
     case [corner]
       when 0 then 0
       when 1 then (([possible votes]-[total votes])/[possible votes])
       when 2 then (([possible votes]-[total votes])/[possible votes])
       when 3 then 0
     end
else
     case [corner]
       when 0 then (([possible votes]-[total votes])/[possible votes])
       when 1 then 1
       when 2 then 1
       when 3 then (([possible votes]-[total votes])/[possible votes])
   end
end

case [rectangle]
when "no vote" then
case [corner]
when 0 then 0
when 1 then 0
when 2 then 1
when 3 then 1
end
when "yes against party" then
case [corner]
when 0 then 0
when 1 then 0
when 2 then (([voted yes]-[yes with party])/[total votes])
when 3 then (([voted yes]-[yes with party])/[total votes])
end
when "yes with party" then
case [corner]
when 0 then (([voted yes]-[yes with party])/[total votes])
when 1 then (([voted yes]-[yes with party])/[total votes])
when 2 then ((2*[voted yes]-[yes with party])/[total votes])
when 3 then ((2*[voted yes]-[yes with party])/[total votes])
end
when "no with party" then
case [corner]
when 0 then ((2*[voted yes]-[yes with party])/[total votes])
when 1 then ((2*[voted yes]-[yes with party])/[total votes])
when 2 then ((2*[voted yes]+[no with party]-[yes with party])/[total votes])
when 3 then ((2*[voted yes]+[no with party]-[yes with party])/[total votes])
end
when "no against party" then
case [corner]
when 0 then ((2*[voted yes]+[no with party]-[yes with party])/[total votes])
when 1 then ((2*[voted yes]+[no with party]-[yes with party])/[total votes])
when 2 then 1
when 3 then 1
end
end