Info Clarity

Sometimes words can replace data

noreply@blogger.com (David Boyle) — Mon, 26 Dec 2011 15:07:00 +0000

I've used a lot of word clouds recently. But I think of them as charts really, since they are still pretty faithful to the underlying data. The size of the word is proportional to the number of times that word is in the data set. Simple.

But reading a cool data visualization book I came across this. Really it not based on 'data', but it's interesting his words and their location on the page conveys such a lot of information. Perhaps some good, well placed words can replace the need to chart actual data?

http://creativeroots.org/2011/03/italy-infographic-map/

Little sparklines in hospital notes

noreply@blogger.com (David Boyle) — Mon, 07 Nov 2011 06:55:00 +0000

Simple, easy to read, but really powerful. Nice little sparklines spotted in the papers from the 20 week scan my wife just had. Cool little chart like this should be everywhere!

And by the way, it's a boy!

Fixing ranking charts?

noreply@blogger.com (David Boyle) — Fri, 09 Sep 2011 06:18:00 +0000

I've been playing with ranking things in charts recently, rather than charting their actual values. There are some clear benefits to ranking things:
- sharp focus on the order
- clear sense of best / worst
- simple to grasp for non-technical users
- we avoid having to squint to see small differences on a chart

... But there are some big problems, also:
- no sense of real-world distance between points
- no sense of whether anything on the chart actually scored 'well'

So for example the music charts are simple to grasp. And the concepts of 'number one' and 'top ten' are often discussed, we have no sense of whether number ten sold anything like as many as number ten. And whether number one sold a lot or a little.

I tried to take the best of both worlds with this visualization of the popularity of dance music in a few countries. It seems to do the trick for me. Let's see if it works for anyone else :)

Org charts that really show what's going on

noreply@blogger.com (David Boyle) — Fri, 09 Sep 2011 06:08:00 +0000

Who knew there were so many ways to draw an org chart? And that they could express so much about how relationships work in a company? I love it!

From http://www.bonkersworld.net/images/2011.06.27_organizational_charts.png

My hard to read report is good for you

noreply@blogger.com (David Boyle) — Tue, 26 Apr 2011 15:21:00 +0000

So much for trying hard to bring data to life and make it easier to use. Some research generated a raft of articles recently that essentially argue for making things less accessible and harder to read!

"If a piece of information is very easy to process (e.g., in large Arial font), we may be over-confident in how well we've retained that information—and more likely to skim it—, while information that's presented in a more challenging fashion (e.g., small Comic Sans MS font) forces us to read more carefully and think more deeply about the material."
http://m.lifehacker.com//5795705/improve-your-learning-and-remember-things-better-by-switching-font-styles

Now I feel terrible for wasting so much time trying to make data clear and simple to use. I should have stuck to complex charts and equations all along :)

Standing on the shoulders of giants

noreply@blogger.com (David Boyle) — Thu, 31 Mar 2011 07:16:00 +0000

Waiting for a plane to take off on a cold, rainy heathrow runway just
now, a thought occurred to me about one reason I love data and helping
people use data. It's about the process of learning from others and
helping others learn from you.

My degree was in pure maths, which helped structure this process.
Since it is without any distractions from real world assumptions and
complications, I'm not sure there is a more structured way to build
knowledge than pure maths. There are a couple of key ways pure maths
works that are pretty helpful in the real world, also.

The first is the use of other people's work. In maths, people spend
their whole lives proving that something is true. They get the result
named after them, but what they also get is the knowledge that their
work will help others to go further. Because in maths, once something
is proved you can take that piece of knowledge and make use of it. You
can take one result that one person has proved and use it to help you
prove something else. There's no need to re-prove it. No need to even
really understand how the first person proved it. You can just take it
and make use of it. No progress would be made if you had to revisit
things that had already been proved.

The second is that people write things down and share them. Results
and logic are written down using a common language, so that people can
look them up and understand them. The process would fall to pieces if
people didn't write things down, didn't explain how they reached their
conclusions, didn't share their work and didn't use a common language.

They're pretty simple things, right? No magic. No wizardry (although
if you read pure maths it does look kinda like wizardry :). But yet in
business we often don't work like this.

In business we often fail to learn from others. We start from scratch
all the time. We forget to ask ourselves 'what do we already know
about this problem that we can build on?'. And when we work something
out, crunch some data or draw some conclusions, we fail to write it
down and share it properly. Documents and presentations capture only
part of the story, the rest of which floats in the head of the author
or in the room during the presentation and then is gone.

So here is to learning from others, building our knowledge and not
re-creating the wheel. And here is to writing things down, using a
common language and sharing our work. Here is to standing on the
shoulders of giants and to helping others to stand on our shoulders.
Sometimes its the simple things that help us to make a big difference.

Some data challenges (part 1)

noreply@blogger.com (David Boyle) — Sat, 04 Dec 2010 15:32:00 +0000

Some data challenges (part 1)
Originally uploaded by David Boyle

I've been thinking a lot recently about all of the challenges I've seen day to day in making data useful. There are many more than I can capture here, but I thought I'd highlight some of the key ones I've most enjoyed wrestling with.

I'll introduce them here, and then I'll do a post on each challenge in the coming weeks.

1. What data should we actually make use of? I'll argue that focus is vital, and less is more.

2. How to compare? When you have multiple sources of data, how do you bring them together to add value, without confusing?

3. How do you combine data with people's skills and judgement? This is rarely talked about, as we often pretend that data drives decisions. In reality, it's almost always head + heart, so let's embrace that and work out how to optimize it!

4. How do we check skills & judgement with data, and yet support skills & judgement, so we don't blindly follow the data? Sometimes an analyst's job is to play down the data to help people to be confident enough to inject their skill & judgment.

5. Clear objectives. Data has so many uses. My live is decision-making, but there are so many other uses (good, bad and indifferent). How do we get clear about what we're actually trying to use data for?

It's gonna be fun to think about these in the coming weeks. I hope it's interesting to you! As always, I'd love thoughts, ideas and stories in the comments.

Using data day-to-day

noreply@blogger.com (David Boyle) — Sun, 17 Jan 2010 13:03:00 +0000

I'm sat next to our new son Emmett in the special care baby unit. While he sleeps I'm Reading his file. It's an impressive volume.

It's something I'd be really excited to see in any of the organisations I've worked with over the last few years. For a number of reasons:

1. It puts data at the heart of decision-making. Understanding, explanation and planning is all based on data. This isn't a goal or something that is talked about. It's the only way it's done. And if the facts change, people change their minds.

2. Data is gathered by and used by the same people who do the day-to-day. Unsophisticated but appropriate tools are used to capture data (see the chart below: hand-written on paper. No systems integration, no IT projects). The people gathering and presenting data are not analysts! They don't outsource it to 'data experts'.

3. Data has a 'so what' attached. See the jaundice chart below. When Emmett went above the dotted line, he got a sun bed. If he had gone above the dark line, he would have got a more serious intervention. Two observations below the dotted line and he comes of the sun-bed. Impressive, eh? Data, rules, diagnosis, intervention. All EASY to use and easy to explain.

4. Process compliance. Data and analysis isn't aspirational or done to impress the bosses. It's a process that is actually followed. Processes are the way people think and act.

5. Plans produced as a result. That's right. Despite being under-staffed and over-worked, the staff make time to plan. Objectives are identified. Interventions are linked to the objectives. And this is all written down and used to guide action. I can't count the number of organisations I've worked with that skip this step. It's thought of as a specialist skill that requires planners to rent a suite in a hotel for a week with flip-charts and brainstorming. But Emmett's ten page plan produced within hours of his birth shows that planning really can be a natural part of doing your job, and needn't be a big deal.

How was this achieved? I think there are a few lessons that I'm taking away with me.

1. Training. Proper training, with objectives, training materials and assessment. They make time for it out of their day job. They value it. And it's done properly, with proper training, trainers and resources.

2. Process compliance. People must follow processes. They must be assessed against this.

3. The right tools. We must give people the right tools to gather and make use of data, without complex systems and without needing to outsource it to analysts.

(Sent from iPhone by Emmett's cot in Queen Elisabeth Hospital. And before anyone asks, he's going to be just fine. Hopefully we'll have him home today. Yay!)

What does one trillion dollars LOOK like?

noreply@blogger.com (David Boyle) — Wed, 25 Mar 2009 12:57:00 +0000

It's amazing how difficult it is to make sense of large numbers. I think they really do need to be compared to something to make sense. I loved an approach that the New York Times took a couple of years back.

The following is another great example from an email forward that I just got. ***** All this talk about "stimulus packages" and "bailouts"... A billion dollars... A hundred billion dollars... Eight hundred billion dollars... One TRILLION dollars... What does that look like? I mean, these various numbers are tossed around like so many doggie treats, so I thought I'd take Google Sketchup out for a test drive and try to get a sense of what exactly a trillion dollars looks like. We'll start with a $100 dollar bill. Currently the largest U.S. denomination in general circulation. Most everyone has seen them, slighty fewer have owned them. Guaranteed to make friends wherever they go.

A packet of one hundred $100 bills is less than 1/2" thick and contains $10,000. Fits in your pocket easily and is more than enough for week or two of shamefully decadent fun.

Believe it or not, this next little pile is $1 million dollars (100 packets of $10,000). You could stuff that into a grocery bag and walk around with it.

While a measly $1 million looked a little unimpressive, $100 million is a little more respectable. It fits neatly on a standard pallet...

And $1 BILLION dollars... now we're really getting somewhere...

Next we'll look at ONE TRILLION dollars. This is that number we've been hearing about so much. What is a trillion dollars? Well, it's a million million. It's a thousand billion. It's a one followed by 12 zeros. You ready for this? It's pretty surprising. Go ahead... Scroll down... Ladies and gentlemen... I give you $1 trillion dollars...

(Look at the little guy in the corner at the left. And notice those pallets are double stacked.) So the next time you hear someone toss around the phrase "trillion dollars"... that's what they're talking about.

I only read the Beano

noreply@blogger.com (David Boyle) — Sun, 25 Jan 2009 20:23:00 +0000

Amongst many fascinating stories in Malcolm Gladwell's Outliers, chapter eight has one that brings back a strange memory for me. And one that should remind us to always question whether the data we are shown really supports the conclusions we draw from it.

Consider the following table of scores achieved by kids in Baltimore public schools across 1st-5th grade. (The test referred to is the California Achievement Test, but that's not important to the example.)

What conclusions might we draw from this? I think we might reasonably start to think that Baltimore public schools were failing low income pupils. They start off with only a slight disadvantage from their moneyed peers (32 poits), but end school significantly under-performing them (73 points).

You get a very different story if you look just at what happens during the time pupils were in school. Karl Alexander tested pupils at the start and at the end of every school year, enabling him to measure how many points they gained while actually in school. Here are the results:

Now it seems that, if anything, schools are of more benefit to poorer kids. Across the five grades, during the school years, they gaines 189 points, while the wealthy kids gained only 184 points. The difference between the first table and the second lies in how many points the pupils gained or lost during the long summer holidays:

And if we know this, we know that the story isn't about education at all. Its about what happens in the school holidays. Poor kids on average neither gain nor loose points over holidays. But richer kids consistently gain. So by the end of the school period, they outperform their peers.

So the analytical lesson here is to always be careful that the data you are using really supports the conclusions you are drawing from it. Ask yourself: If you cut the data differently, might it tell a completely different story? If so, give it a shot.

And what of the Beano you might ask? I have an odd memory in primary school of my class gathering around the teacher after the long summer break one year. He asked us to each pick one book we'd read over the summer and tell the class about it. One by one the pupils in my class told their classmates about one of the books they had enjoyed. When the teacher pointed to me, I had to ask whether a comic counted, since that was all I'd read over the summer.

1.4 million New Yorkers in DC for Inauguration!

noreply@blogger.com (David Boyle) — Tue, 20 Jan 2009 08:07:00 +0000

... or so it might seem if you take polling results seriously.

In a recent poll, 9% of New Yorkers said they were planning to head to DC for the event. There are about 16 million adults in the New York area, suggesting 1.4 million people planned to make the trip. The lesson here is that there are some things you shouldn't use polling for!

From pollster.com: One problem with a question like this one may be that it lends itself to social desirability bias. As we know, citizens tend to over-report the extent to which they will (or did) vote in elections. In a similar way, some respondents may be proclaiming that they will attend the inauguration when they don't have any real intention of going. They may do so because they hear of so many others who are attending and they feel as though it is something they should be doing as well.

This reminds me of another article pointing out something else you shouldn't use polls for: asking people whether they were at historic events.

From Political Animal: I remember reading, years ago when I lived in Miami, that a significant percentage of the population of South Florida believes they were in attendance for the famous Dolphins-Charges playoff game in 1982. That's impossible, of course, since the capacity of the Orange Bowl was only about 75,000, and the population of Miami-Dade is in the millions, but locals remembered the game so fondly, they'd fooled themselves into thinking they actually saw the game in person. It's similar to the phenomenon of the number of people claiming to have been on hand for Woodstock in 1969 -- more people believe it than could have possibly shown up.

You also shouldn't use polls to ask people whether they did things that turned out to be a very, very bad idea:

Again from Political Animal: [in a recent poll] only 33 percent of respondents admit to having voted for the guy twice, while 52 percent said they'd never voted for him at all. If that were actually true, of course, Bush would never have had the chance to run the country so firmly into the ground that people are now pretending they never liked him

... so what does that leave us that polling IS good for?

Sometimes its not cool to gather data

noreply@blogger.com (David Boyle) — Thu, 08 Jan 2009 12:14:00 +0000

There are some occasions where you need to keep your data gathering quiet!

From the excellent XKCD.

I also love this one. I wish I'd have thought of it!!

Excel 2007 is good for something, after all

noreply@blogger.com (David Boyle) — Mon, 08 Dec 2008 14:52:00 +0000

Ok, so there are actually a few cool features in there. Its still annoying to get used to, though!

One feature I've found pretty handy recently is the conditional formatting enhancements. They really let you visualise data quickly and easilly. Great stuff.

There are a wide range of new options that are just a button-press away:

Here's an example of what I've been using it for:

Much better than looking at a table of numbers, I'm sure you'll agree!

Interest rates racing down

noreply@blogger.com (David Boyle) — Fri, 05 Dec 2008 17:05:00 +0000

There was a nice little chart in the Wall Street Journal yesterday showing clearly how central banks are racing interest rates towards zero.

(From WSJ via Paul Kedrosky)

Florence Nightingale's eye for info clarity

noreply@blogger.com (David Boyle) — Fri, 05 Dec 2008 07:49:00 +0000

I just got around to reading Science News' article on how Forence Nightingale was a pioneer in using novel and innovative charts to present data. Apparently she went to great lengths to do so in order to convince Queen Victoria of the need for social change. She was worried that without the clear presentation of data in charts, Queen Victoria's eyes would glaze over as she scanned statistics and tables of data.

Her most famous chart is an enhancement to what we would call a pie chart. It shows the number of deaths each month by their cause.

Each month is a twelfth of a circle. Months with more deaths are longer, meaning the area of each month shows the number of deaths. You can see that during the first part of the war, the blue wedges (disease) aremuch bigger than the red ones (wounds) or black ones (other causes). After March 1855, when the Sanitary Commission arrived, the blue wedges start becoming dramatically smaller.

From Science News:

The conventional way of presenting this information would have been a bar graph, which William Playfair had created a few decades earlier. Nightingale may have preferred the coxcomb graphic to the bar graph because it places the same month in different years in the same position on the circle, allowing for easy comparison across seasons. It also makes for an arresting image. She said her coxcomb graph was designed “to affect thro’ the Eyes what we fail to convey to the public through their word-proof ears.”

Some argue that a bar graph would have made her point more dramatically, though. One of the peculiarities of Nightingale’s circular presentation is that the deaths are proportional to the area, not the radius. Since the area of a circle is pr2, the area is proportional to the square of the radius rather than to the radius itself. This difference tends to de-emphasize the contrast between the small areas and the large ones. (In an early version of this diagram, Nightingale didn’t catch this distinction and drew the graphic incorrectly, with the radii proportional to the deaths. She quickly corrected her mistake.)

It seems to have worked. The 830 page report she wrote lead to massive changes in hospitals and by the end of the century, Army mortality was lower than civilian mortality.

Spotting fraud in numbers

noreply@blogger.com (David Boyle) — Sun, 30 Nov 2008 19:57:00 +0000

I remember reading about Benford's law years ago with fascination and thought I'd share it. Such a fun use of maths in the real world. Here's one application:

Dr. Theodore P. Hill asks his mathematics students at the Georgia Institute of Technology to go home and either flip a coin 200 times and record the results, or merely pretend to flip a coin and fake 200 results. The following day he runs his eye over the homework data, and to the students' amazement, he easily fingers nearly all those who faked their tosses.

Smart, eh? Its all because people don't know enough about how numbers really work and so can't fake data convincingly. The first thing that people get wrong when faking data is assuming that each number 0-10 has an equal chance of being used. They do not. In the real world, numbers are much more likely to start with a '1' than any other digit.

From "The First-Digit Phenomenon" by T. P. Hill, American Scientist, July-August 1998

Benford's law can be used to predict the frequency of numbers. As you can see fro mthe above, it matches closely to real-world data sets. It predicts that '1' is the most likely first digit, then '2' less so and so on and so on.

When you see the analysis of fraudulent data sets, it really comes to life:

From "The First-Digit Phenomenon" by T. P. Hill, American Scientist, July-August 1998

From the same article:

Benford's law can be used to test for fraudulent or random-guess data in income tax returns and other financial reports. Here the first significant digits of true tax data taken by Mark Nigrini from the lines of 169,662 IRS model files follow Benford's law closely. Fraudulent data taken from a 1995 King’s County, New York, District Attorney's Office study of cash disbursement and payroll in business do not follow Benford's law. Likewise, data taken from the author's study of 743 freshmen's responses to a request to write down a six-digit number at random do not follow the law. Although these are very specific examples, in general, fraudulent or concocted data appear to have far fewer numbers starting with 1 and many more starting with 6 than do true data.

Back in 2005 I worked with a team of organisers who would report the number of doors that each of their teams had knocked on each day. I thought it would be fun to see how that data compared to Benford's law. Overall, you see that it doesn't look like people were being honest:

Buit its not all bad news. It looks like some were more honest than others:

Why keep data?

noreply@blogger.com (David Boyle) — Sat, 22 Nov 2008 15:02:00 +0000

Not the best case study in the world, but a fun one:

I was bored at work seven years ago today. I can tell that because I was just looking through old emails and seven years ago today I was emailing Seb with an 'audit of recent text messages'. Here is who I was text messaging seven years ago:

What fun to have that data and be able to look back - not at who I remember being in touch with - but who I was ACTUALLY in touch with. There are some names on there I had completely forgotten about! One good reason to gather data and keep it laying around, I guess. I'll probably dig it out again in another seven years and then I'll really appreciate it!

(If I hadn't swapped phones so often I'd have kept all text messages since then and I could do some trend analysis. But technology has sadly not made this possible, but I'm starting again now with my iPhone and this helpful site)

(As shown in a previous post, there is a neat tool to do this kind of analysis on your Gmail, but I'm not enough of a nerd to be able to get it to work. Rubbish.)

Common BNP names

noreply@blogger.com (David Boyle) — Wed, 19 Nov 2008 17:29:00 +0000

Since someone leaked the British National Party's membership list to the internet, I thought I'd take a look.

Here are some common first names:

And some common last names:

No surprises there then!

I couldn't find a Singh or Patel. Odd.

Misleading conclusions

noreply@blogger.com (David Boyle) — Wed, 19 Nov 2008 10:31:00 +0000

I'm currently working on advertising strategy and someone just forwarded me a document arguing that when a major brand 'goes dark' on advertising for a long period, it significantly damages the brand. But I drew the opposite conclusion from the document!

The article is by an organisation with -to put it politely- a vested interest in keeping the advertising dollars flowing and so there are a few misleading things in there. But I thought one misleading conclusion was particularly fun. Here it is:

Note that they chose to not visually display the 'no change' result, because it is by far the biggest segment and does not help their case! As a result their tag line is plain misleading: The most likely thing to happen when a brand goes dark is actually that there is no impact on the measures they show! That should be the headline message: "brand measures are unlikely to change when you go dark."

And the chance of going dark and having no impact or of an impact being favourable is 72%-76%! That is pretty good odds, I'd say. I'm willing to bet that we've done smarter analysis than the competition and so if we were to do that we'd get even better odds ...

How to avoid supermarket waste

noreply@blogger.com (David Boyle) — Thu, 13 Nov 2008 12:09:00 +0000

I saw another article today on how supermarkets waste too much fresh food. I thought I'd post a quick chart showing an easy way to reduce waste: buy more stuff!

Slow selling products lead to waste. They always will. So look for products that don't sell very well and buy more of them!

The following chart shows what waste might be expected for fresh food, at different sales rates. Slow selling fresh food (2 singles per week in the chart) might be expected to usually waste more than 150% of what you sell. So for every two you sell, you waste more than three! (And some products waste way more than this, if they are very short life, or if they must be ordered in large cases from suppliers for example.)

You'll see that this decreases massively as sales increase slightly. Go from sales of two to four singles per week and your expected waste falls below 100% of sales. Increase the weekly sales to eight and you're back at a reasonable level: expected waste of about 20% of sales.

So if you are worried about the ammount of food that supermarkets waste, simply look out for the slow selling products and put a couple in your basket. a couple of extra sales will make all the difference to supermarket waste.

When you are lost, any map will do

noreply@blogger.com (David Boyle) — Sat, 08 Nov 2008 12:56:00 +0000

I was playing around with a really cool and easy to use analysis tool for Excel, Analyse It. And I thought I'd show how easy it is by running an old data set through it. So here is some quick analysis of an old political campaign I worked on. Seriously, this took about 30 minutes total.

We won the race overwhelmingly. In fact we won every precinct in the city. One thing the tool allows you to do is compare groups of data. Here is our vote share by city ward, with each blob representing a precinct:

This type of chart can be created with about three button presses. It shows we won big in our home ward (yay!). It also shows the distribution of precincts. They were pretty bunched together, but it might be interesting to look at the outliers. Why were they so different?

We did an ok job of predicting where turnout was going to come from. I can tell this by running a regression of our predicted turnout by precinct and the actual turnout by precinct. The tool does this very simply, with no specialist knowledge required:

This graph also takes about three button presses to generate, and comes with a neat set of stats telling us that we did an ok (not great) job of predicting turnout by precinct. But then we should have done, since historical turnout by precinct was widely available, and the best predictor of future behaviour is past behaviour.

Now here is the interesting bit: we were crap at predicting how people would actually vote. I should stress that this was a Democrat versus Democrat election, so it is a notoriously difficult thing to call. But either way, we didn't do a good job! Here is our categorisation of precincts in to how we thought they would vote in advance versus how they actually voted:

You have to say that's pretty bad. In the precincts we called for our opponent, we actually did better than those that we thought were too close to call. Its good to know that we did better in the precincts we thought would support our candidate. But in those precincts there is a really wide range of support. They could probably have been better broken out in to different categories.

We did try to break them out further, but without much luck. Here is a more detailed look at what vote we thought we would get versus what we actually got:

We did a good job of predicting the very best precincts, but beyond that group, all of the other five groups we decided on behaved very similarly.

We also broke the precincts out that we thought our opponent would do well in. That also didn't turn out to be a good predictor of the actual vote:

Pretty depressing really!

Data and targeting has played a huge part in many political campaigns. That is particularly clear from '08 and organisations like Catalist have played a huge part in this. In politics I had the pleasure of working with a fantastic group of people who would honestly evaluate their own work, but ...

... usually after elections finish, candidates' political campaigns close their doors and everyone goes off to the next campaign, tired and often ready to forget about the campaign. There is no will to evaluate what worked and no money to fund evaluation. And besides, the people who did the analysis often don't want to evaluate for fear that their analysis that they were paid handsomely for wasn't actually helpful after all. This is a tragedy.

Even if predictions aren't that helpful, having lots of data on a campaign makes people believe. Everyone has confidence in a well presented graph and they feel that those extra phone calls and those extra door knocks are being well targeted. In reality, I often believe that when you are lost, any old map will do.

However if there were more evaluation after the effect, perhaps campaign analysis would improve quicker and it would more often actually help the campaign to win, rather than just boosting morale.

Knowing where you were

noreply@blogger.com (David Boyle) — Fri, 05 Sep 2008 10:15:00 +0000

I'm loving my iPhone. Today I'm loving it mainly because it adds another dimension to the data that I can gather: where things take place. For example, I use InstaMapper to track where I am in the world. Here is my journey home from work: And here is a *live* map of the most recent time I ran the program:

&amp;amp;amp;amp;amp;amp;lt;br&amp;amp;amp;amp;amp;amp;gt;

Another great feature of the GPS is that it automatically geocodes photos. For example, I took regular photos here to track a recent trip to LA. Here a section from the drive to Heathrow through London: [I can't seem to embed the actual map, which allows you to click on a dot to see the photo. But it is live here] Such fun to have that data and be able to look back on where you were at a particular time, and what you saw while you were there. I love it.

Analytics? You can't touch this!

noreply@blogger.com (David Boyle) — Wed, 20 Aug 2008 22:58:00 +0000

Even MC Hammer thinks its cool. (Courtesy of Juice Analytics)

The difficulty and danger of tipping

noreply@blogger.com (David Boyle) — Fri, 01 Aug 2008 10:10:00 +0000

This blog is about how you can analyse information in the world to bring clarity and make better decisions by doing so. This post is about some thoughts on how difficult that sometimes can be! We all like to be nice and to reward people who are nice to us. So why is working out the right tip so difficult? Ideally you should be able to use tips to signal both good and bad service. I like the standard and well understood assumption in New York that 15-20% tip is expected on restaurant meals. I tip 0% or 5% to signal truly terrible service. 15-20% to signal service that was about what I expected and 25% or so to signal that they exceeded my expectations. I have a nice system that I'm happy signals what I want it to ... ... however, even in this simple case, I very much doubt that most waiters take my tip and really know what was meant by the signal! For example:

When I tip 5%, am I bring deliberately unkind? Do I even know that 15-20% is expected? Am I just being 'cheap'?
When I tip 25% am I signalling good service or am I just being flash with my cash? Did I think that was a normal tip?

So even in places we are familiar with, I believe that signalling what you want to signal with a tip is usually difficult. When considering how much to tip, I think you need to balance the following pieces of information at the very least.

There is a lot of information there that you need to make a good decision!!!

First, you need to base your tip on the tip that the person is expecting from you. This varies by industry (restaurant workers yes, shop workers no. Why?), type of establishment (don't tip in a McDonalds restaurant, do tip in somewhere with white table cloths. Why? Do tip in hotels, not in B&Bs. Why?), location (by city and by country. Why?) amongst other factors.
If they are expecting 15% then you need to go to 20% or 25% to show appreciation. Is it ok to go below what they are expecting to signal bad service?
Is the tip already included in the bill? We have all made the mistake of adding a tip on only to realise on the way home that it was already in the bill and we ended up tipping twice!
Why tip as a percent of the bill? Does a waiter in a high class restaurant need four or five times the dollar value of tips that a waiter in a more downmarket place gets?
What wage is the person paid? Can they live on their salary, or do they genuinely need the tips to make a decent wage? If so, the tip is not really a tip. Its really part of the bill. If its really part of the bill, then the service would have to be pretty bad to justify you lowering the tip from their expectation. (You might argue that doing so would be akin to challenging the price of the actual goods / service you received.)

When away from places and industries that we know, it is very hard to find out what is expected as a tip. And as such, it is very hard to deliberately signal and to avoid accidentally signalling! Ask any two people and they will give you different answers. Some examples:

London taxis are, perhaps, the most expensive in the world, but offer perhaps the best trained drivers in the world. So what I consider reasonable for a tip probably doesn't mean much to the driver! I have lived in London for eight years and I'm not quite sure what the cab drivers actually expect by way of a tip!
Do you tip maids at hotels? I spend a lot of time in hotels with work and I can't claim back such tips so I don't leave them. Am I the only person in the hotel to not tip? I honestly don't know. One hotel I stay in a lot has a more local feel. I know the staff and so I feel bad not tipping. But I keep forgetting, so every now and then I leave a substantial tip on the bed with a thank you note. Hopefully this makes up for the times when I don't tip? [Someone recently told me that they tip maids in hotels as an incentive not to steal their stuff or clean the toilet with their toothbrush. That is another concept not included in my diagram: tipping as 'protection money' to avoid being treated badly!]
Do you tip the guy who takes your bag to your room in a chain hotel? How much? I have no idea. I try to carry my own bags to avoid the uncomfortable moment when they pause to see if I will tip and when I fumble through my wad of local currency I got from the airport ATM on the way and desperately try to calculate the exchange rate. Sometimes they insist on showing you to your room and its rude to not accept! Do you tip then?

In less developed parts of the world, it is even more difficult to know what and who to tip. And our inclination to be generous can easily be dangerous. In places you are not familiar with, I think the following is the minimum required to calculate an appropriate tip:

For a start you are probably guided only by your guide book's view of what is 'expected'. Ask any two locals and you will get a different answer! There is the dangerous inclination to be more generous in local terms than you otherwise might be. "Why not just tip five dollars to the boy that helps carry your bags? Its nothing to me!" It is dangerous because you may well be giving that boy the equivalent of a week's wage for a local factory worker. In situations like this you create huge incentives for people to get close to tourists to get tips, rather than to learn skills and get more traditional jobs. Just think what this does to the local economy! But in such a situation you face the choice of looking stingy to the boy who is holding his hand out and to the people who are watching you decide what to give him, or being seen to be a big spending generous person. So its tough to do the right thing and give a tip that is reasonable in local terms rather than one that is reasonable in your terms. This also creates a whole industry of people whose very purpose is to do some small act and then guilt foreigners in to giving them a tip for it. In Marrakesh you will see the boys on the street who are seemingly stood around to help tourists lost in the maze of Souks. Show even the smallest glint in your eye of being lost and one will pounce. Even if you tell him you don't need help and don't tell him where you want to go, he will walk in front of you to your destination and when you get there, he will very vocally demand a tip of a few dollars for taking you there. They are a nuisance created by the lure of what we see as small tips. Who can refuse a few dollars of tip to a kid who has his hand out and is beginning to be an embarrassment to you by insisting that you owe him a tip for showing you the way? What do you think? Am I over-analysing? Is there a simpler way to ensure you tip the right amount without all the worry, above?

Visualising the wedding data

noreply@blogger.com (David Boyle) — Sun, 20 Jul 2008 20:49:00 +0000

Another quick post on the wedding. This time to show one of the benefits of keeping your data in one neat spreadsheet: it allows you to easily play with it! Here is a simple treemap visualisation of the data in IBM's ManyEyes that you can play with: Here is a breakdown of the guests by their relationship with us: And here is a quick look at where people travelled from: Nerdy, but fun!