cookin'/relaxin'

Designing Structured Stories

2016-03-11T15:31:00.001+00:00

This is a written version of a talk I gave at the Interaction 16 conference on the 2nd March 2016, it was also published on Medium.

Stories are everywhere, from newspapers to TV drama, fact to fiction. By deconstructing stories into fundamental, structured building blocks we can present them in new and powerful ways.

We can make experiences that vary the length of the story, let you explore it from different directions, summarise and recap it, change the media used to present it, adapt it to what you already know or personalise it to make it more relevant. And we can create truly responsive content that adapts to your device and situation.

Through two case studies at the BBC, mobile news and drama serials, I’m going to look at how we took a data-driven but user-centred approach to structuring and designing story experiences.

The primary building blocks we use in our work are storylines, key moments and people. The storyline might be the epic journey of a character, the events in a refugee crisis or a love triangle. The essential moments and events in these storylines could be the bits that people remember most fondly, the key to understanding or the jumping off points to other stories. Yet the key people or characters are almost always the most important part of a story.

This is the story of my work at BBC R&D on what I (and others) call structured stories; it’s about what you could make with them, why structure is important and what we’ve learnt doing it.

The structure of stories

My story of this work starts here, with a book — Everything Bad is Good for You by Steven Johnson. It’s about the increasing complexity of modern popular culture. And I was particularly interested in this diagram of plots in American TV crime drama. At the top is Dragnet, from the 60s, where one episode was a single narrative thread, similar to Starsky and Hutch from the 70s. Then it starts getting more complex with Hill Street Blues. Finally it shows The Sopranos which has many substantial parallel plot strands in one episode. If you watch any critically acclaimed TV drama from Scandinavian noir to The Wire to Game of Thrones you’ll probably have noticed this. This started me thinking about what we could do with these structures

This is Kurt Vonnegut’s sketch of the Cinderella story. The y-axis is the happiness of the protagonist, Cinderella. You can see that she goes to the ball, loses her glass slipper and then is found by the prince and they live happily ever after — where the graph tends to infinity!

Luckily lots of clever people have have already thought a lot about stories. Much has been written on the shape of plots, three acts and five acts. The classic five act structure consists of the exposition, the rise, the climax, the fall and the denouement. If I was really good then this article would use that form.

But it’s often not just one thing after another in stories. Memento, the Christopher Nolan film, plays with time. One strand of it is ordered chronologically (the black and white bit), the other goes in reverse (the colour bit), and the two are interleaved in the film (see this diagram). In fact several Nolan films play with time and telling and I’d suggest that lots of media increasingly does this .

A story model

So it seems generally agreed that stories do have structure. One Russian school of literary criticism defined two things that seem particularly relevant to us — the fabula is “the raw material of a story”, what happened, and syuzhet is “the way a story is presented”, the narrative.

In our work at BBC R&D we look for how to do scalable, reusable solutions using a solid backbone of data. We developed a data model for stories with three layers.

We have the world of the story: the protagonists, people, places and relationships.

On top of that, the events: what happens in the story.

And on top of that, the narrative: or how the story is told.

(Paul, who worked on this, has a similar model described here)

Cards

A small diversion from stories. There’s been a recent design trend for cards and designing things smaller than pages. This quote from Paul Adams of Intercom gets to the heart of it

"In a world of many different screens and devices, content needs to be broken down into atomic units so that it can work agnostic of the screen size or technology platform."

To make proper responsive content we need re-usable atomic units of content. And to make them properly re-usable we have to add structure or we won’t know how and when to re-use them.

So if we atomise stories into re-usable components then structure and label these components, we can re-combine them to make new things in new places.

Stories in fiction and reality

I’m going to look in more detail at two kinds of stories — drama and news, fiction and reality — and how we used some of these story principles to make some things.

The Archers is the world’s longest running soap opera — it’s 65 years old this year. It’s about a farming community and is broadcast every day on the radio in the UK. This is it’s card index.

It tells you things like what happened in 1967, who married who, who’s got a piano in their front room and what the cows are called (yes, really). It’s used by the writers so that they don’t get anything wrong and are consistent. There are a lots of listeners that will notice. But also to give them inspiration for new storylines. There was an old Windows database but that stopped working and so our team at BBC R&D designed a new web-based structured continuity database.

The point is, there is already structured data about the stories that the BBC tells. But this was so huge it was a bit daunting to start working with so we worked with a slightly shorter radio drama.

The Story Explorer

In 2015 we started working with Home Front, a BBC radio drama serial charting life on the home front during the First World War, who were nice enough to let us play around with their programme. We took the first two series and using lots of spreadsheets we split the episodes up into the storylines, events or scenes, characters and places involved. For each scene we also chopped out the audio from the programme and wrote a quick summary so we had a self-contained “atom” of media for every event.

Remember that bit about the increasing complexity of modern popular culture? Boxsets and the impossibility of missing an episode have meant there is increasingly complex drama on TV and radio. Jason Mittell has written a whole book on this. We wanted to use this data for those times when you’re watching (or listening to) a drama series and think “Who was that?” or “What just happened?” or maybe you can’t remember what happened in the last series or the last episode. There was a space to make something to help explore TV and radio drama and solve a real problem for people.

A colleague even joked that his mum would love a “Remove post-modernism” button on their TV that just re-ordered everything back into a chronological timeline!

This is the Home Front Story Explorer — you can try it here or watch an intro video here. It’s not intended to be a replacement for listening but it’s useful if you want to quickly catch up or remind yourself what happened, or get an introduction to drama that’s new to you.

You can choose the storyline you want to catch up on and then listen to each scene. Or you can just skim through it and read the bits that interest you. Plus you can see which characters are in each scene and find out more about them. You can read much more about it here.

We piloted it last summer and it went down really well with the audience. Some were new to the drama and used it to get into the story, some just wanted a recap on bits they’d missed or forgotten before the next series started and some even used it as a single-purpose radio — listening to single storylines for hours.

We’re now trying to develop this concept across TV drama — designing it to be re-usable & scalable for all dramas and beyond, maybe even for reality TV.

Structuring the news

We then decided to apply these principles to news and look beyond the conventional news article, which hasn’t really changed since modern newspapers were invented.

Our research showed that some young people found it hard to get into BBC News. They found it to be be quite inaccessible and missing essential contextual information. And if you think about it there’s often lots of assumed knowledge in the way news is written. What was the Berlin Wall? What is an interest rate? Why is there a migrant crisis?

This is the model we’re been using for news, which has been in development for a number of years. You can find the full storyline ontology developed by the BBC and others here. Storylines are made up of events and each event can be linked to people, organisations, places and even other storylines. As you can see, it looks awfully similar to the drama model. And the BBC doesn’t just do news in one way; we do it online, on TV, on radio and these can all be linked in to this structure.

This example is the story of Alexis Tsipras’s resignation and the Greek debt crisis. Obviously, as with most news, it’s complicated and subjective. Where do stories start or end? Where do they intersect? What is the truth? It’s really interesting and a bit harder than drama and fiction.

It turns out that the processes of journalism have quite a lot of innate structure that we could use. Journalists are taught to use the 5 Ws — who, what, where, when & why. The inverted pyramid of journalism suggests you should lead with the most essential info at the top, getting less and less important to the end. It was originally so editors could cut a column from the bottom to fit the space available in the newspaper.

The BBC used to run a text information for TVs, it was called Ceefax. Amongst many other things it had news stories, all made up of four paragraphs. This structure made it onto BBC News online because journalists would write for both and, despite the demise of Ceefax, you can still see it now in the first four paragraphs of most articles on BBC News.

Atomising news

We’ve had a go at re-inventing the news article for mobiles using structured stories. Through several iterations of prototyping and testing we discovered that people liked having quick, skimmable summaries of stories, particularly on mobile. But they also wanted the option to be able to go deeper and get more information on the aspects that interest them.

Every story is structured as a storyline — with key events, people and places. The initial view of the story gives you a summary on one page, so you can just skim the the key events. But you can expand any of them and dig deeper — into longer-form writing, correspondent reports, social media or video. Plus we have pop-up definitions available of the key people, organisations and places for that extra context if you need it.

Elsewhere

I’ve found that structured stories are everywhere when you look. Here’s some obvious examples I found during this work.

Writers often use structure — this is a diagram showing how long-running stories for a comic series were planned out.

This is a screenshot from Arcadia — a novel available as a book and an app. In the app you can switch between the interlocking storylines at many points.

Games are often structured stories — it’s most obvious in interactive fiction, like 80 Days. Social media could even be described as the structured stories of people’s lives.

Fragmentation

But why use structured story data for any of this? It goes back to the earlier quote. By atomising our stories and fitting those atoms to a data model we create re-usable bits of stories that we know when and how to use and then can put together again in different ways. These structures give us the ability to adapt our stories to different screens, platforms, contexts and experiences. Whether that’s watches, conversational user interfaces, AI or things without screens. We could even adapt stories to you — maybe drop the bits of the story you already know or change the language and tone.

I think this is particularly important for media organisations. There are always more and more new things, devices, platforms and places to get stories, media and news. From Google AMP to Facebook Instant Stories, from Line to Vine, from chatbots to Snapchat. Rather than create more and more stuff for each of these new platforms piecemeal as they appear, if we structure our stories into small, reusable pieces then we can efficiently use them again and again. That makes structured stories very powerful.

Though, I might add, not necessarily a wholesale replacement for long form content or content carefully crafted for the channel.

Design patterns

These are some patterns I’ve noticed throughout this work.

Building blocks

Where should you start if you want to build structured stories? I’d suggest you identify the main protagonists, what happens to them in their story and how you want to tell the story. Then try using the model of the story world, the events and the narrative.

Beginnings and endings

Knowing when to start and stop stories is important because you want things to be the right length for people, but also not hide the complexity or full story.

We tended to try to present our stories at an appropriate length for the format & then used two techniques. We added navigation in the Story Explorer to take people from one story to another. In Atomised News we have the concept of entire storylines embedded within events — they describe themselves well enough for their context but they also link out to the full storyline elsewhere.

Order of telling

As we’ve already seen, many storytellers deliberately change the order in which information is revealed to make the story more engaging. So we don’t always want to present events in chronological order. You might expect that at least news articles would be chronological, but they frequently jump between what just happened and what happened previously.

So we separate the events from the narrative and enable them to be presented in many ways according to the storyteller’s desire, or even the user’s preferences. Examples of ordering include chronological timelines, reverse chronological blogs, an author’s narrative order or even algorithmic timelines (as seen in Facebook).

Skimming and digging

Different people will want to consume the stories in different ways. Sometimes we might be telling the whole story, sometimes just a summary.

In the Story Explorer we flagged certain scenes as essential for a storyline, enabling us to provide a shorter “highlights” experience for each story. In Atomised News the expanding spine provides both an overview and the full story for those that want it.

Mixed media

The most appropriate representation of a story may change depending on the person or the situation or the device. But we can hang multiple media representations of the story off the structure and then choose which to use. These representations might be different lengths of content, different media types or even different tones or styles.

Challenges

In this work we tried to balance using a re-usable data model that could underpin scalable experiences while also designing for user needs. This meant that we were iteratively developing the UX, the stories and the data simultaneously and this required a lot of talking and sketching and compromising to make sure the whole team were in agreement and at the same level of understanding.

We needed tools to make structured stories, but we didn’t want to be putting extra things in the way of writers and journalists. The Archers continuity database is a good example of a tool which is part of the existing workflow and actually useful to writers, while creating structured story data as a by-product. And we are looking to make this process even easier by automatically processing existing media — we’re currently investigating the feasibility of machine-parsing programme scripts to (semi-)automatically create story data.

Finally, we tried to make sure that we kept the stories intact, didn’t mess with them too much and let the storytellers do their job — I think we succeeded.

Reading list

Everything Bad is Good For You by Steven Johnson

Into The Woods: How Stories Work and Why We Tell Them by John Yorke

Complex TV: The Poetics of Contemporary Television Storytelling by Jason Mittell

The end of apps as we know them by Intercom

A three-part model for understanding stories by Paul Rissen

The future of news is not an article by NYT Labs

Events, causation, articles, reports, stories, repetition, insinuation, supposition and journalism as data by Michael Smethurst

The Home Front Story Explorer prototype by BBC R&D

Acknowledgements

Paul Rissen, Michael Smethurst, Zillah Watson, Chris Sizemore, Jacqui Maher, James Jefferies, Andrew Wood and the rest of the team at BBC R&D.

Doctor Who illustration by Yoon Bahk.

2015-08-16T15:02:00.004+01:00

I don't write very frequently here but I do collect some recent interesting links in an occasional newsletter. If this works, you can subscribe to it here:

Or if that doesn't work, try here: https://tinyletter.com/tristanf

Train departure times

2014-08-25T14:57:00.002+01:00

I was wondering if it was a coincidence that my daily trains home from Waterloo are mainly at 0, 15, 30 and 45 minutes past the hour. As it is the major terminus for all the trains leaving London to the south-west then the timetables might well be derived from this station and may use nice numbers for neatness.

trains.im has all the appropriate data. For an arbitrary weekday I scraped and tidied the data, then plotted the departure times, by minutes past the hour, for all the trains leaving London Waterloo during a single day.

There's no obvious pattern, just two peaks at 20 and 50 minutes past (of 52 and 51 trains departing respectively). It also turns out that the train departures are pretty evenly distributed for each quarter of an hour.

In conclusion, if you don't know when your train is leaving Waterloo it's best to get there for 20 past the hour or 10 to. And there are no trains at :01, :04, :08, :11, :17, :19, :26...

Types of prototyping

2014-05-28T12:49:00.000+01:00

For future reference. I'm extracting this from my contribution to the team's weeknotes the other day. I've been collecting some links and articles about approaches to prototyping, here are three particularly good things to read...

Pretotyping
The "Pretotyping" site has lots of interesting reading - it's about working out as early as possible whether people want something, rather than "prototyping" which they define as working out if we can build something. The authors describe a number of interesting approaches from creating an ad for something to determine interest, to using humans behind the scenes to power something all the way to MVPs.

Flavors of prototypes
Marty Cagan recently wrote about the different flavours of prototypes that he sees. He identifies feasibility prototypes to address whether things are technically feasible, low-fidelity user prototypes or wireframes to think through ideas, high-fidelity prototypes to test usability and communicate the idea and live-data prototypes to prove whether an idea works for real.

What do prototypes prototype?
Finally, Houde & Hill's classic paper "What do prototypes prototype?" identifies three dimensions of prototypes. The "role" dimension answers questions about how it is useful to people, the "implementation" answers questions about how it works and the "look & feel" answers questions about how it feels to use it. The "integration" prototype explores a balance of the three dimensions.

Poster by Andrew Nicolaou from Houde & Hill

A prototyping process
Do some or all of these...

1. Make low-fidelity prototypes for users, answering questions about the role of the thing and what it's used for. And for helping you think through ideas - i.e. thinking through doing. This is where the "Pretotyping" approach would come in.

2. High-fidelity user prototypes. These will answer questions about the look & feel and usability. They also help you communicate and sell ideas.

3. Technical implementation and feasibility prototypes. Working out whether you can actually build this and what you need to make it work.

4. Integrated, live data prototypes that combine the results of these initial prototypes and test the idea out for real.

See also "A Guide to Building Prototypes"

Archives, algorithms and people

2013-11-20T12:06:00.002+00:00

Or..."how we put the BBC World Service radio archive online using machines and crowdsourcing"

A talk given to the UK Museums on the Web conference, London, November 2013.

One of the major challenges of a big digitisation project is you simply swap out an under-used physical archive for its digital equivalent. Without easy ways to navigate the data there's no way for your users to get to the bits they want. We recently worked with the BBC World Service to generate metadata for their radio archive, 50,000 programmes from over 45 years. First using algorithms to generate "good enough" topics to put the archive online and then using crowd-sourcing to improve the data.

Throughout 2013 BBC R&D have been running this experiment to crowdsource improvements to the metadata that we automatically created. In this prototype people can search and browse for programmes, listen to them, correct and add new topics.

This talk describes how we went about this and what we've learnt with this large online media archive - about understanding audio, automatically generating topics and crowdsourcing improvements to the data.

Our team had a challenge - a big radio archive, sparsely described, to put online. The BBC typically puts archives online by editorially curating them, so only a small subset is exposed. But we thought we would try putting a whole archive online, as efficiently as possible. We built a prototype and over the past year we've run an experiment.

We had an opportunity to work with the archive of the World Service English language radio service. They'd digitised their archive as they were having to move out of their historic home at Bush House, into the new Broadcasting House. The archive contained about 70,000 radio programmes from over 45 years. Not everything though, there are no live news bulletins, they weren't recorded, and it's (mainly) English language.

The graph shows how this archive is distributed from 1947 to 2012. The spike starting in the 90s is where the BBC started to use digital technologies record things. And stopped recording over older tapes.

The digitisation process created very high quality digital audio of all the programmes, but artefacts of the digitisation process (and indeed earlier archiving) meant that the metadata describing the programmes was sparse - often missing fields or having incorrect data. And if there's no data describing the programmes then no-one will be able to find them. It’s a danger with a big digitisation project - that you simply swap out an under-used physical archive for its digital equivalent. Without easy ways to navigate the archive there's no way for your users to get to the bits they want. And to navigate the archive you need data.

That was our challenge. And we wanted to demonstrate how to create the data needed to put a massive media archive online using algorithms, linked data and crowdsourcing.

We needed to generate data primarily from what we did have - the digital audio from the radio programmes. We used CMU Sphinx, an open-source speech recognition toolkit, to listen to every programme and convert it to text.

Speech recognition can be very good, particularly when trained on a single speaker but on these radio programmes, with varying recording qualities and many voices and accents from around the world, it really struggled for accuracy, and we ended up with lots of pretty noisy transcripts. But we didn't need accurate transcripts, just some good metadata.

Our team in BBC R&D developed algorithms that could reasonably reliably extract tags or keywords from these noisy transcripts. We use Linked Data to provide unique tags (e.g. to disambiguate Paris, France from Paris Texas), to help this topic extraction, to relate tags to one another, and ultimately to link to elsewhere on the web. We actually use DBpedia, a data version of wikipedia, as our reference. So every tag in our archive is linked to a wikipedia page and for every programme, even if there was no metadata to start with, we typically generated 20 new tags.

This was a lot of media to process - about 26,000 hours of audio. Doing this audio transcription and topic extraction would have taken around 36,000 hours on a single computer. But using the cloud we could do this all in parallel and we processed it all in 2 weeks at a cost of around $3000.

The automatically created tags weren't always correct and we couldn't go through them all to check but our hypothesis was that they were good enough to bootstrap an online archive for people to use and listen. And then we could ask those people to help correct and add to these topics - to crowdsource the problem.

The prototype can be found at http://worldservice.prototyping.bbc.co.uk and this short video introduces it.

The homepage shows some featured programmes or you can search. From the results you can filter by decade and then listen to the programme. You can view the automatic topic tags and then vote them up if they're correct, or down if they're wrong. And you can add a new tag, corresponding to a wikipedia page. It's registration only, but easy to sign up at http://worldservice.prototyping.bbc.co.uk.

It's worth noting that the original metadata wasn’t created for public consumption so it can be pretty perfunctory. And a note about images: we didn’t have any to start with but people expect images on the web, so we use the tags to choose images from Ookaboo, a repository of public domain and Creative Commons images. Users can select alternative images if the don't like the default.

Users can also directly edit the programme title and synopsis, to correct spelling mistakes for example. It uses a Wiki-model to track changes and has a small admin interface for us to "clear" them.

Here you can see the list of tags for a programme, some automatically generated, some with user votes, and some added by users. Rather than the wiki-model on the text edits we use a voting model here, more like reddit.

One last bit of the prototype, this is very cool, we are also doing speaker identification and segmentation automatically on some programmes. We can recognise distinct voices within, and between, programmes. The only thing we can't do is automatically identify who it is that is speaking, so we ask users to label the voices. This shows a magazine programme (From Our Own Correspondent) that the algorithm has divided into the different contributors, users can then label those voices. And these names then propagate across the archive to wherever else that voice was heard.

To recap:

Start with a media archive with little metadata
Process it with machines in the cloud
Use that data to create an online experience
Get people to use it and improve the data

...and we want to feedback these improvements to the machines to help them learn. For example, we can look for tags that are often voted down and then look for patterns in them.

There are 70,000 programmes in the archive.

36,000 are listenable (the audio is available for all users) and 34,000 aren't available, either because of rights issues or because the actual audio is missing but a record was created for the programme.

Originally there were 1 million machine generated tags.

Currently (November 2013) around 3000 registered users.

71,000 edits, where an edit is some kind of action from a user - either voting on a tags, adding a new tag, labelling a voice, editing a synopsis or voting on an image.

Of those there were 70,000 tag edits, formed of 57,000 votes and 13,000 new tags.

1000 synopsis edits.

Overall around 36% of listenable programmes have been listened to at least once (30,000 total listens). And about 21% of listenable programmes have an edit, at an average of 9 edits per programme.

Listeners even sent in 4 "lost" programmes that they had recorded off-air

The machine-generated tag quality looks OK by eye and the crowdsourced tags are good, I almost never disagree with them. But this is hard to answer objectively. When is a tag correct? In whose opinion? And even harder - when does a programme have a “complete” set of tags? And as these programmes can be about almost everything, and our tagging space is wikipedia it's a very large and sparse space of data. There's also more work to do to analyse what kind of tags are added and whether people listen to the programme before tagging. Tagging a 30 minute radio programme is a bit different to tagging a painting or photograph.

There's been a surprising amount of synopsis editing - spelling changes, adding comments, adding credits... One person particularly likes adding episode numbers!

Who has been adding data? One person has astonishingly done 30% of the edits. Ten people have done 70% of the edits and 10% of the total users have done 98% of the work. Other crowdsourcing studies have shown that typically 10% of users do the majority of the work. The internet also has a 1% rule - 1% create, 9% modify and 90% just view. About half of our users have done at least one edit. But that doesn't really tell you that much, do they stick around? We use “Active users” - a term borrowed from wikipedia - an active user is someone who has done some edit action in the last 30 days. Our active users are currently around 2% of the total.

We've noticed particularly groups of people using the prototype. It started with a large community of radio drama enthusiasts who were cataloguing all the drama and plays. And more recently some Frank Zappa fans appeared. So do you only need 100 people to do this? Well, without the archive of programmes and the prototype drawing people in we wouldn't have found the "right" 100 who care enough to help.

Some pictures we drew using the data, giving an idea of some of the archive contents and activity.

User tags, clustered by the programme they're attached to.

Users, clustered by what they listened to - "The Lobster".

As we’ve got Wikipedia-mapped tags we can look for programmes about places.

Links from current news events being talked about on the BBC News channel (via the subtitles)

back to programmes in the archive.

And my favourite of the things we found. From 1957 - The last broadcast from the BBC Danish Service. “Entirely in Danish”.

The quality of original metadata was mixed and we've significantly improved it with algorithms and crowdsourcing, adding semantic topics to all of the programmes. We couldn’t have created a decent online archive otherwise, we just didn’t have the data.

It's also efficient. The initial research & development cost was less than our estimated cost of professionals tagging everything. And this technology cost is a one-off and re-usable. It becomes cheaper the more times we use the technology and it can be used for any media, radio and TV, with people speaking in English.

We think we're a bit stuck in middle of some different crowdsourcing approaches. If you know Galaxy Zoo and its crowdsourcing projects - they are generally designed to be very task focused with particular targets - to get the most out of their volunteers. Our prototype was more of a browsable archive with crowdsourcing features added. Maybe closer to Wikipedia. We don't know what's right, and it would be interesting to compare the two approaches. And we've tried both a wikipedia "last edit wins" approach and a reddit voting approach.

There are some things we still don't know.

How good are the tags? Like I said, it's difficult to measure objectively

How much volunteer effort do you need? It depends. How big is the archive? How much data do you need per item? How good does the data need to be?

And ultimately, when is your data good enough?

You can register for the prototype here and try it for yourself.

You can read more on our website and blog.

A number of components of the system have been open-sourced and can be found on github.

And in this project we did a lot of work to manage the processing. We found this so useful that we're turning it into a generic platform, called Comma, for anyone to analyse their media, and for any computer scientists to run their analysis algorithms.

Analysing media in the cloud

2013-06-26T13:03:00.001+01:00

This is a talk I have at the IBC London conference yesterday on cloud technology. I described two projects that BBC R&D is involved in that use cloud computing for processing media. The first is a case study showing how we used cloud computing to efficiently process a very large archive of media and generate metadata, and the second part is about how this led to us to think about abstracting a service out of it, leading to a general purpose cloud service for analysing media.

Analysing media in the cloud from Tristan Ferne

These are my talk notes.

Two case studies

I'm going to talk through two TSB-sponsored projects that the BBC is involved in that use cloud computing. The first is a case study showing how we used cloud computing to efficiently process a very large archive of media, and the second part is about how this led to us to think about abstracting a service out of it, leading to a general purpose cloud service for analysing media.

ABC-IP & World Service

2 years ago, Working with an SME, Metabroadcast, we got funding from the TSB in their "Metadata" competition to look at ways to automatically generate metadata for media archives, aiming to link archives together and to unlock value by finding content that was previously hidden.

The BBC World Service archive

We needed an archive to work with and we got an opportunity to work with the archive of the World Service English language radio service. They'd digitised this archive as they were having to move out of their historic home at Bush House, into the new Broadcasting House in central London. It's about 50k programmes from 45 years. The graph shows how the archive is distributed over time - from 1947 to 2012. The spike is where we started to use digital technologies in the 90s.

The missing metadata

The digitisation process created very high quality digital audio of all the programmes but the metadata describing each programme wasn't so good. Artefacts of the digitisation process (and indeed earlier archiving processes) meant that the metadata describing the the programmes was sparse - often missing fields or having incorrect data. And if there's no data describing the programmes then no-one will be able to find them. A danger with a big digitisation project is you simply swap out an under-used physical archive for its digital equivalent. Without easy ways to navigate the data there's no way for your users to get to the bits they want.

Listening machines

So for our project we needed to generate data primarily from the digital audio. We used CMU Sphinx, an open-source speech recognition toolkit, to listen to every programme and convert it to text.

Noisy transcripts

Speech recognition can be very good, particularly when trained on a single speaker. But on these radio programmes, with varying recording qualities, many speakers and accents from around the world, it can really struggle for complete accuracy, that's an example. But we didn't need accurate transcripts, just accurate metadata so we looked at topic extraction.

Extracting topics

Our researchers developed algorithms that could reasonably reliably extract topics from these noisy transcripts, so , for instance, we could say that a programme was about music, Benjamin Britten and Gustav Holst. We use dbpedia (a data version of wikipedia) to provide unique topics (e.g. to disambiguate Paris, France from Paris Texas) and to relate topics to one another. For every programme, even if there was no metadata to start with, we could generate some topics.

Processing archives in the cloud

We had a lot of data to process, the 50,000 programmes was about 26k hours of audio
So, obviously, we used cloud computing. Doing this audio transcription and topic extraction took the equivalent of 36k hours of a single low-processing-power computer. But using the cloud we could do this all in parallel and we processed it all in 2 weeks at a cost of around $3k for the processing and data. To manage the parallel processing we build KiWi, a message-based system to co-ordinate the various tasks and computers.

Machines + People

But we weren't finished. Although we had automatically created topics for every programme they weren't always completely correct and we couldn't go through them all to check. Our hypothesis was that they were good enough to bootstrap an online archive for listeners to use. And then we could ask those listeners to help correct and add to these topics - to crowdsource the problem. This diagram shows the hypothesis:

Take an archive
Process it with machines in the cloud
Use that data to create an online experience
Get people to use it and improve the data
...and feedback these improvements to the machines to help them learn

World Service prototype

This is the prototype we built featuring the 50,000 WS programmes. You can search and browse for programmes, listen to them, correct the topics and tag them with new topics.

Here you can see the list of tags for a programme, automatically generated, with users votes alongside
And (as aside), we are also doing speaker identification and diarisation (splitting into who's talking) automatically. This shows a magazine programme divided into the different journalists. We are currently running this experiment to crowdsource improvements to the metadata that we automatically created using our automatic transcription, topic extraction algorithms and cloud computing. So far we've had over 60,000 additions or corrections to the metadata. You can register to try it out there... worldservice.prototyping.bbc.co.uk

COMMA

That was our first project, automatically generating metadata for a massive radio archive, something we probably couldn't have done without cloud computing (or at least it would have been very expensive). It set us thinking about whether this would be a useful service for other archives in the BBC, and indeed elsewhere. And beyond that, how this would apply to other metadata generation or extraction services for media. We realised that much of what we'd done here could be applicable to lots of things and useful to lots of people.

At that time TSB were running a competition for innovation in cloud services so we found two partners (Somethin' Else - a content design and creation company and Kite - an internet development consultancy) to develop a proposal, and we were again successful in getting funding for COMMA (working title).

Analysis

COMMA is a service to support all kinds of analysis of all kinds of media, these might be things like:

Topic or keyword extraction from text (or audio!)
OCR of text in scans or images
Automatic summarisation
Sentiment or mood analysis of text or other media - is that person happy or sad? Is that tweet sarcastic?
Speaker identification (given some voice samples you can identify where they speak elsewhere) and diarisation (splitting audio up into different speakers)
Music identification (like Shazam)
Face or person recognition in images or video (e.g. for actors or celebrities)
Object or place recognition in images (e.g. track the moving car, recognise St Paul's Cathedral)
Automatic segmentation of video (e.g. recognising shot changes or "scenes")
Automatic subtitling

All of which could be very useful for automated production tools, for reducing costs in common operations, finding things in archives, new audience-facing services or new opportunities we haven't thought of yet

Problems with media analysis

But there are a few problems with using the current state-of-the-art in media analysis. Some of these algorithms can be incredibly computationally intensive, especially across very large collections - that's where cloud computing comes in. They can be complex to set up and use, certainly there'll all be different. They can be hard to integrate into your systems; they have different data formats, software libraries, APIs... Therefore it's hard to compare different services and it's difficult to know exactly what's out there, and what's possible.

And for someone who provides analysis technologies, similar things apply (particularly if your a small company or university, where many cutting edge technologies come from) and it can be hard to provide a service that anyone could easily use

Making media analysis easy

COMMA is intended to be a cloud platform to make media analysis really simple. Content owners will be able to easily upload media and then choose what they want to analyse or what metadata to extract. Algorithm providers can upload their algorithms (in a way that safeguards their IP) and set their costs
And then the platform manages everything else

The comma marketplace

The system includes...

A secure mechanism to allow content makers to upload their content
An counterpart system to allow algorithm providers to securely host their code
A marketplace where customers can choose from a range of open-source and commercial algorithms
A cloud-based processing system to do the heavy lifting
A monitoring system where customers can see how things are progressing
A payment system to manage billing

Our hope is that COMMA will act as a marketplace making it really easy for algorithm providers sell their technologies and for media owners to analyse their archives and collections, unlocking additional value, previously unseen content and creating new opportunities for services and applications

Our goal is to spend the next 2 years building a prototype metadata extraction platform that will be secure, scalable and simple to use. The intended outcome of the project is a fully-tested and 'market-ready' prototype shaped by the requirements of key content owners and algorithm providers. You can read more about COMMA on the BBC R&D website.

A short introduction to metadata

2013-06-26T12:44:00.003+01:00

This is a talk I gave at the Wellcome Institute in the Digitisation Doctor workshop on April 15 2013. It is a brief overview of what metadata is and why it's useful, particularly with respect to archives.

An short introduction to metadata from Tristan Ferne

These are my talk notes:

What is metadata?

Data, or information, about something, usually structured. Data that describes, identifies, classifies. In this case, data that describes the thing/content you're archiving. It might be designed for humans, it might be designed for machines to read.

What is metadata for?

It's to identify things, to find things, to re-use things, to link things to other things. If your content is text then you can search for words in it but if it’s an object, or a video or a picture then you need to describe it somehow.

What does metadata look like?

Example photo

http://www.flickr.com/photos/tristanf/6093308933/in/set-72157627652249662
Unique identifier (filename vs Flickr ID on the web)
Creator - me
Date and time - created by digital cameras
Location - might be from GPS, this was created by me, it's on Reydarfjordur, Iceland
Usage - statistics on how many people have viewed it
Title and description - written by me
Tags - created by me to describe the photo - it's an aluminium smelter, it's dusk, there's a bird, a fulmar
Camera info - created by digital cameras, things like make and model of camera, ISO, f-stop, focal length, image resolution
Copyright / licensing - Creative Commons Attribution licence

XML example

This is what some of that looks like in a computer readable form - something called XML. It's consistent labels and formats for how those fields are described.

Example of a programme

A BBC programme on the web, Masterchef: http://www.bbc.co.uk/programmes/b01rrmyp
Unique ID, that 8 character string is our unique internal ID, and added to the bbc.co.uk gives a globally unique ID
The brand title
Series, episode
When and where it's on
Duration
A description
Credits - who made it, who was in it

This data’s also XML, something called RDF

Catalogue record

This is an actual catalogue record from the BBC archive for clips of a helicopter from a programme from 1972
http://www.flickr.com/photos/nicecupoftea/5364762501/

Standards are good

Using a standard means someone's already gone to the trouble of defining what things you might want to describe, systems / software probably already support it and it's interoperable - other people can understand it, and you can understand other peoples' data, though maybe with the drawback that it doesn’t describe exactly what you want to.

Dublin core

I don’t know your fields or collections so I can’t really recommend standards but Dublin Core is a broad set of defined metadata that is widely applicable. It defines the fields and how each should be written. The Simple Dublin Core Metadata Element Set (DCMES) consists of 15 metadata elements:

Title
Creator
Subject
Description
Publisher
Contributor
Date
Type
Format
Identifier
Source
Language
Relation
Coverage
Rights

Where it the metadata?

Is it "in" the object? I.e. you can embed data in digital photographs - EXIF - or digital audio - MP3 tags. It’s probably fairly limited how much you can embed but you can’t lose it! Or it’s in a database that is somehow linked to the object. Or both.

How to create metadata

Once you’ve decided what you want to describe about your objects you need to create it. Some data may come from the digitisation process or you may already have detailed catalogues. But it's quite likely you're going to have to generate some or all of it, at least entering it into a database somehow.

Three options

1. Experts / archivists / cataloguers / librarians
2. Volunteers / crowdsourcing
3. Computers

Crowdsourcing

Asking volunteers to help create metadata. Often online tools, often specially designed to encourage people to contribute as much as possible - game-like, leaderboards, tasks... You might have heard of some of these...

Galaxy Zoo

This is Galaxy Zoo, volunteers were asked to look at pictures taken by computerised telescopes and classify whether the shapes of the galaxies. In its first phase 80,000 volunteers classified 10 million images

Your Paintings Tagger

And this is the BBC and Public Catalogue Foundation’s Your Paintings Tagger. It’s an online collection of all the paintings in the UK in public ownership. It asks people to describe each painting in number of ways.

Machines and algorithms

Thirdly, you can use machines and algorithms to automatically generate metadata, this could be:

Keyword and topic extraction from text
Machine translation
Speech recognition from audio
Speaker identification and diarisation
Face detection in images or video
Object and scene recognition

There are free and paid services on the web, open-source software and companies that sell this technology.

Putting it on the web

If you want to put your archives on the web, to make them available to anyone then hopefully from what I’ve shown it’s obvious that you need metadata to make the website browsable and searchable

the web = internet + links (http) + documents (html)

We, the BBC, think it's important that when putting things on the web we publish the metadata there too. It means things are more findable. It means other people and organisations can build on top of our data and content. This means we can link things to other places on the web, and vice-versa...and we're such a large organisation it even helps us link between our own sites

Linked data

We use linked data, also known as the semantic web. Broadly it's about describing things online so that machines can understand them, uniquely identifying things and being unambiguous about them - strings vs things. It means we can start to automatically make connections between things. There are linked data vocabularies for...people, social media, commerce, events, radio and tv programmes, music and much more.

World Service radio archive

Finally, a little plug for a project I’m working on but I don’t really have time to go into detail about here
We’ve been working on putting the World Service english language radio archive online. There was a 3 year digitisation process that digitised around 50,000 programmes from the last 60 years as high-quality audio. Unfortunately much of the metadata was sparse and sometimes incorrect - Programme title, short description and a date, if we were lucky. Typically the BBC would put editorial effort into selecting, describing and publishing archive content online but this wasn’t practical on an archive of this scale - nearly 3 years of continuous audio.

Automatic transcription example

So R&D got involved. We ran speech-to-text algorithms across all those programmes - took us a couple of weeks, a couple of thousand pounds, using a cloud-computing service. These generate very noisy transcripts but our researchers developed advanced keyword / topic extraction algorithms that were pretty good.

The prototype

The topics were still not perfect but they were good enough to build this website where anyone can explore and listen to the programmes. And we use crowdsourcing so if users want to help they can vote on whether the automatic topics are correct or not, or even add their own topics. If you’d like to explore this archive, or know someone who would be interested:

http://worldservice.prototyping.bbc.co.uk

How I remember what to watch

2011-02-22T16:15:00.000+00:00

This is a list of lists. All the places where I keep lists of TV and radio programmes that I want to watch or listen to.

A text file

This text file (well actually a note in Notational Velocity and Simplenote) is a list of TV programme that have been recommended to me; by word-of-mouth, by Twitter friends, by blogs or by newspapers. These are often US dramas and most likely to be things I buy as a DVD boxset. I just note them down when I can (on my phone or laptop) and the “x3” next to Fringe is because it’s been mentioned 3 times. Some of these (e.g. Supernatural) are ones where I want to remember to buy another series when it’s released, some are new.

A Freeview PVR

I use my PVR quite a lot to record broadcast programmes for watching later. Most programmes recorded on there are through a series link (it records all episodes of a series) and I think all are TV despite it being able to record radio. It’s fairly new, I only decommissioned my 10(?)-year old TiVo recently, so there’s plenty of disk space on there. One anomaly is that BBC 4 doesn’t work, something to do with Guildford transmitter, so the PVR records BBC 4 programmes but they are unwatchable. In which case the programmes sit on the list acting as a reminder for me to watch them on iPlayer.

A dog-eared Radio Times

Yes, i buy the Radio Times. I have a ritual where I generally try to skim through it on a Saturday morning and then fold page corners down when I see something interesting. Then later I will go through these marked programmes and set up the PVR to record or maybe check it in an evening if I’m looking for something to watch. At Christmas I might even draw a ring around programmes.

iTunes

Most of my radio listening is Radio 4 and most of it is live and at the weekend - their regular schedules and my memory helping me do that. All my podcast subscriptions are managed through iTunes though I never manage to listen to many, they’re more likely to be saved for a holiday. iTunes does some management by notifying you that you haven’t listened to a podcast for a while and stopping it updating (the little exclamation marks next to subscriptions). I tend to ignore this and just tell it to keep downloading. I also download one-off radio programmes, mainly discovered through the Radio Times, but I can’t tell you how I do that!

So what?

It's somewhat fragmented with different tools and workflows for TV, DVDs and radio, adapted to meet my particular needs and technology. And I'm sure I'm not typical in any of these habits. Also interesting is where these recommendations came from in the first place - word of mouth from friends and Twitter friends) or articles that happen to mention a programme (“...Niall Ferguson’s Civilization begins on 6 March on Channel 4”).

What about you, what do you do?

Garden birds in 2010

2011-01-30T13:08:00.002+00:00

This is a record of what birds we've seen in our garden during 2010. Last year's report can be found here.

Breeding blue tits (up to ten or so). They used our nestbox.
Coal tits
Great tits
Occasional flocks of long-tailed tits pass through.
Four or five greenfinches visit regularly since I set up a sunflower seed feeder.
A pair of bullfinches
Two chaffinches
A pair of blackcaps
A few dunnocks
Robins
Several blackbirds
Woodpigeons
Collared doves
A single thrush
One regular nuthatch
Two goldcrests
Two wrens
One greater spotted woodpecker, actually on the tree next door.
And a rare redwing, brought by the snow.

And for this weekend's Big Garden Birdwatch I spotted 4 blue tits, 4 green finches, 2 blackbirds, 2 dunnocks, 2 robin, 2 bullfinches, 1 woodpigeon, 1 great tit, 1 wren, 1 nuthatch and 1 magpie. And 2 long-tailed tits flew over.

My 2010

2011-01-07T15:57:00.005+00:00

Sitting on a rock on the Bjäre peninsula eating lunch in the sunshine, September 2010

It was my first proper year back in R&D and in the Prototyping team and I'm now very definitely a producer and product manager. Certainly more people and less computers. I started writing weeknotes in February, I'm not sure how compelling reading they are but they're a good discipline. And we had a number of interesting projects - an RFID-enabled toy, a trending music site, a Twitter links site, a second screen experiment and lots of storytelling work including the Mythology Engine.

I replaced my old MacBook Air with a very similar looking new Macbook Air, but with less moving parts.

I moved all my note-taking into Notational Velocity (plus SimpleNote for the iPhone).

I didn't blog here very much but I did finally write up some notes on How to build prototypes.

I visited Copenhagen, Southern Sweden, Grenada and Lyon, managing that with two train trips and only one flight.

I took lots of photos with my new Panasonic GF1.

I read lots of books, some on my new Kindle, but listened to less music and sadly played almost none.

Apparently I ran 200-odd miles, including a couple of 10k races.

And I played rather too much Wesnoth on my iPhone on the train home.

Skyr

2010-12-15T15:02:00.003+00:00

Skyr is a traditional Icelandic yoghurt, made from sheep’s milk. I’ve finally found a source in the UK, the excellent Scandinavian Kitchen on Great Titchfield Street, just around the corner from my work. they occasionally stock skyr and they apparently have a skyr email list if you want to be notified.

The original, traditional skyr is really thick and is prepared by mixing it with milk to the desired consistency. The version shown here is ready-mixed, but not flavoured or sweetened. And actually comes from Denmark. But I’m assured it tastes right. So, if you want to prepare some skyr…

1) Put the appropriate amount into a mixing bowl.

2) Add sugar to taste and mix thoroughly.

3) Serve in individual bowls, top with a little soft brown sugar and a dash of milk, or even cream, around the edge.

(or serve with bláber compote)

Technologies for creating online relationships from real-world actions

2010-07-02T14:42:00.004+01:00

Some recent research into technologies that create online friendships or links using real-world interactions, typically proximity. Any more in the comments please.

Poken

Poken is a physical USB key-like object where you touch other Pokens to share a business card or contact details then sync it over USB To your computer. It seems to use a proprietary wireless technology. If you want to know what’s inside, take a look at Hacking the Poken.

Bump

Bump is an iPhone and Android app that lets you exchange information by “bumping” your phones together. But what’s notable is how it works. From their FAQ…

“Q: How does Bump work?
There are two parts to BumpTM: the app running on your device and a smart matching algorithm running on our servers in the cloud. The app on your phone uses the phone’s sensors to literally “feel” the bump, and it sends that info up to the cloud. The matching algorithm listens to the bumps from phones around the world and pairs up phones that felt the same bump. Then we just route information between the two phones in each pair.”
“Q: No way. What if somebody else bumps at the same time?
Way. We use various techniques to limit the pool of potential matches, including location information and characteristics of the bump event. If you are bumping in a particularly dense area (ex, at a conference), and we cannot resolve a unique match after a single bump, we’ll just ask you to bump again. Our CTO has a PhD in Quantum Mechanics and can show the math behind that, but we suggest downloading Bump and trying it yourself!”

Disney Fairies

Bracelets from Disney that allow kids to become online friends by touching the objects together then returning home and docking them over USB.

“Tink Friendship eBracelets bring girls and Disney Fairies characters together with the mere touch of a band powered by Clickables™ technology. The magic begins with the creation of a Fairy Friendship Kit online at www.PixieHollow.com where a girl can select her fairy avatar, a special message and a gift, then save it onto her Tink Friendship eBracelet to be shared offline with friends. When a girl touches her band to her friend’s and presses a button, her band will glow to confirm that a Fairy Friendship has been made no cords, no computer, just a touch of Pixie Dust! Just like the girls, their online fairies are friends too and can easily find each other in Pixie Hollow.”

Petimo

A research prototype from Keio University where children are only able to accept online friends in a proprietary online social network by bringing their Petimo toys into physical contact. The toys also feature a screen and a wireless connection to allow other interactions and features.

Cuddly robots aim to make social networks child-safe - tech - 07 May 2010 - New Scientist

Petimo: Safe Social Networking Robot for children (PDF)

Facechipz

Facechipz is a proprietary social network that uses the distribution of physical tokens with unique codes to create online friendships.

“The way it works is: parents register their child on the site for a one-time fee of USD 1. Kids purchase packs of five collectible FaceChipz tokens from select retailers and then register each token online by entering the unique code printed on the chip. Once the FaceChipz are registered, kids hand them out to friends. A receiver goes online and they also enter the chip’s code. The FaceChipz database then confirms the friendship, and the two people are linked.”

Online network for tweens requires offline introductions

A guide to building prototypes

2010-04-16T16:08:00.003+01:00

I’ve been trying to express some of the principles and ideas that I use in my day-to-day work in the R&D Prototyping team and this post has been sitting on my desktop for far too long. Think of it as a first draft and let me know if it’s useful or what’s missing or even what’s plain wrong about it. I’m not going to go into why you would or should prototype, and some of these things obviously apply to all kinds of projects. It is inspired by countless things including 37signals “Getting Real” (if you follow this, you’ll be fine), 43folders, Diego Rodriguez’s Innovation Principles, lots of people I’ve worked with, and more. I’m not sure any of what follows is original, and it’s not comprehensive, but this is my list and I find it useful.

Ideas are easy, building things well is hard.

Build things to surface problems and understand things. Until you build it you won’t truly understand the opportunities or challenges.

Read in and around the problem area. A lot. There aren’t many new ideas and lots of things already exist. This should be obvious but it will stop you wasting time.

Make it simple, then simpler still if you can.

Embrace constraints, they are there to help you and make you more creative.

Define the approximate scope of what you’re going to do, you can always change it later but you’ll have something to work to.

Know when to stop. Make up a deadline if you don’t have one. And reduce scope if you’re not going to make it.

Work in an agile manner, but I don’t really care if you have a methodology or not.

Get T-shaped people, or at least have T-shaped multi-disciplinary teams.

Work together all the time. Try to get your engineers and designers working and thinking together and exploring problems from both perspectives simultaneously. This can be hard.

Talk a lot within your team. Make sure you know what everyone else is doing.

Support team-members’ passions but be pragmatic.

Sketch. On envelopes or napkins or with code. This helps communicate the problem, get a shared understanding of things and get further support for the work.

Make it just good enough to work and look good enough to attract people. But not too polished; the lowest resolution you can get away with.

Tell people about it. Write about it. Communication helps form your ideas and develop your prototype further.

Don’t chase shiny things.

Don’t get hung up on a particular technology. Build it with anything that works, you can refine it later.

Try to make it fun but not frivolous.

Just start, I know it’s hard, but once you’re going it will get easier. And even if your first attempt fails you’ll have learned something.

Update: Having just presented this as a lightning talk, there is now a set of slides up on Slideshare.

The Nature of Technology

2010-03-12T17:05:00.003+00:00

I’ve just got round to writing up my notes on The Nature of Technology: What it is and How it Evolves by W. Brian Arthur. The overall theme of the book is along these lines…

p.2 “Technologies in other words consisted of other technologies, they arose as combinations of other technologies…But it meant, I realized, that if new technologies were constructed from existing ones, then considered collectively technology created itself.”

p.3 “Technologies consisted of parts - assemblies and subassemblies - that were themselves technologies. So technologies had a recursive structure.”
“And every technology, I realized, was based upon a phenomenen, some effect it exploited, usually several.”

On how technologies are combinations of other technologies…

p.21 “If we put these two pieces together, that novel technologies arise by combination of existing technologies and that (therefore) existing technologies beget future technologies, can we arrive at a mechanism for the evolution of technology?”

p.25 “Slowly, at a pace measured in decades, we are shifting from technologies that produced fixed physical outputs to technologies whose main character is that they can be combined and configured endlessly for fresh purposes.”

Talking about domains of technology and Jules Verne’s From the Earth to the Moon he says…

p.74 “You recognize them as mid-1800s because of the component sets they draw from: the iron-place cladding of the craft; the artillery cannon that hurls it into space; the brick-and-wrought-iron structures that house the venture. Such component sets and the way they are used do not just reflect the style of the times, the define the style of the times.”

On insight and the origins of technologies:

p.116 “It comes as a moment of connection, always a connection, because it connects a problem with a principle that can handle it…And it not in the midst of activities or in frenzied thoughts, but in moments of stillness.”

p.128 “Origination in scientific theorizing, as in technology, is at bottom a linking - a linking of the observational givens of a problem with a principle (a conceptual insight) that roughly suggests these, and eventually with a complete set of principles that reproduces these.”

And concludes with…
p.209 “The economy, in a word, is becoming generative. Its focus is shifting from optimising fixed operations into creating new combinations, new configurable offerings.”

p.213 “It is reinforced nonetheless by the qualities of modern technology: its connectedness, its adaptiveness, its tendency to evolve its organic quality. Its messy vitality.”

There’s also a lovely little sketch of how technology has evolved in human history, but you”ll have to buy the book for that. Fairly interesting overall but not as compelling as I had hoped.

Garden birds in 2009

2010-03-07T19:36:00.002+00:00

A little late but here is the list of birds seen in our garden during 2009. Last year's results are here.

A goldcrest
A wren
Long-tailed tits
Blue tits
Great tits
Coal tits
Dunnocks (breeding I think)
The occasional sparrow
Robins
Chaffinches
Greenfinches
Blackbirds
Thrushes
Starlings
Woodpigeons
Collared doves
A pair of greater spotted woodpeckers

And a male and female blackcap have recently taken up residence in the garden over the winter.

Some visualisations of stories and narratives

2010-01-13T15:57:00.011+00:00

Timelines are pretty common visualisations, there’s a lovely history of them here from Cabinet Magazine. My current project is looking at representing narratives and drama online and as part of this I’ve been researching existing visualisations of narratives and stories, some of which I’ve collated below along with my own sketches and thoughts. They’re in approximate order of complexity and we start off with some relatively simple hand-drawn diagrams illustrating general plot features.

The Archers

A hand-drawn diagram from one of the scriptwriters on The Archers radio drama. It shows the intensity of the storyline for each character over the week.

x: time, y: intensity of story, a line per character

Cinderella

Another hand-drawn diagram, this from Kurt Vonnegut in his book, Palm Sunday discovered via this post. He shows the Good or Ill fortune of the protagonist over the duration of the story. The steps up represent the gifts from Fairy Godmother to Cinderella culminating in a plunge at the stroke of midnight when everything changes back again, but then the prince finds here and she lives happily ever after (tends to infinity).

x: time, y: fortune

Tristram Shandy

Illustrations from Tristram Shandy by Laurence Sterne circa 1760. Described by Cabinet Magazine as “indicating the non-linear path of a well-told story; narrative digressions appear as deviations from a straight line.”.

x: time, y: digression

Napoleon’s invasion of Russia

The classic infographic from Charles Minardi, “Carte figurative de pertes successives en hommes de l’Armée Française dans la campagne de Russie 1812-1813” represents the story of Napoleon’s march into Russia and shows location, time and his army size.

x/y: map, width: size of the army, annotations: events and temperature

Now onto some interpretations of plot and episode structures.

Hill Street Blues

Diagrams from Steven Johnson’s Everything Bad is Good for You illustrating the increase in complexity of contemporary television drama and the evolution of multiple storylines. The featured image shows a Hill Street Blues episode with each grid square representing a scene and each row representing an individual storyline.

x: time/scenes, y: storyline

Jazz on 3

Following on from Steven Johnson’s diagrams I drew these a few years ago showing the fractal-like nature of nested events in TV and shows. “The radio show (Jazz on 3 in this instance) can be broken down into segments - the introduction, a discussion of the artist, interviews and a live session. Each of these segments is further broken down (in green) into the individual interviews and the individual tracks.”. This could also apply to TV and radio drama.

x: time, y: scenes

The Archers again

This comes from a project a couple of years ago to build a web app that segmented the Archers radio drama into individual scenes and marked up each one with relevant facets. This diagram illustrates how the episodes are split up and how ongoing storylines would thread their way through the episodes.

x/y: time/episodes/scenes, annotations: facets and storylines

Movie narrative charts

And then there’s this classic from xkcd to “…show movie character interactions. The horizontal axis is time. The vertical grouping of the lines indicate which characters are together at a given time.”. The Lord of the Rings diagram is beautifully constructed to simultaneously show time, character groupings and major locations and events.

x: time, y: character and groups, annotations: places and events

Stories are rendered as TV Programmes

I recently drew this after considering TV drama and how a show is a particular rendering of a story into the form of TV. The conceptual story happens along a timeline with significant plot events marked. These events are then rendered in scenes in the TV programme thereby creating a programme timeline. But sometimes events may be portrayed in multiple scenes, as flashbacks or different points of view for example. And sometimes a single scene on the TV may portray multiple narrative events. An ongoing story arc comprises many events in the timeline from many episodes.

x: time, y: rendering, annotations: connections and storylines

Memento

The film Memento is a particularly good example of using a complicated structure to create a compelling story and Density Design have created this set of visualisations based on it. The example above has slices showing the order of the scenes in the film and then the same scenes re-ordered as they would have been experienced by the protagonist. This diagram is also interesting and shows character involvement as a coloured bar in each scene, ordered both by film- and real-time.

x: time, re-ordered by film or real timeline, colour: b&w or colour scene

Now let’s have a look at some time travelling stories which makes things a bit more complicated…

Doctor Who: Blink

This diagram comes from my colleague Paul Rissen who has been doing some very deep thinking about modelling narratives in data. It is based on events from the Doctor Who episode Blink) that features characters being thrown back in time. The diagram shows various timelines from the show - the timeline from the perspective of the main character, Sally (which is the same as the timeline shown in the broadcast episode), the universe timeline (i.e. how events occurred in linear time and different from the episode) and the timelines of various other characters who time travel in the episode and thus experience events in a different order.

x: time/scene (re-ordered by character), annotations: connections

Time travel from fiction

An original work from Information is Beautiful showing all the time travel in various films and books on a single timeline.

Universe timeline, annotations: time travel

George Bush

Ben Fry talks about these complex hand-drawn depictions by Mark Lombardi of “social/commercial interactions and their hierarchies, and politics” where “…he began to create drawings such as this one to depict the complex narratives he would uncover through his curiosity about anything from failed banks to corruption in government to organized crime.”. This one shows the investments and relationships of George W Bush. An interesting combination of a timeline and complex relationships.

x: time, annotations: relationships and connections

Jules et Jim

Finally, another set of film visualisations from Density Design based the the film Jules et Jim. The one shown above uses the curve of the storyline to represent the feelings and involvement of the three characters. There are lots of others in the set.

x/y: storyline and feelings, colour: involvement

I’m particularly taken right now with the very simple illustrations of the shape of the story from Kurt Vonnegut and I’d like to incorporate something like that into our project. And time travel also gives you a lot more to play with. Have you seen any more story visualisations that you like? Please add them in the comments below.

Books read in 2009

2010-01-11T12:55:00.002+00:00

This is the third year I’ve done this. Overall I read 14 fiction and 17 non-fiction, 31 in total. I reckon that’s about 60-70cm of bookshelf per year, which btw is overfull, any bookcase recommendations appreciated. To save you reading the whole list my favourite books of 2009 were Anathem by Neal Stephenson for fiction, The Ongoing Moment by Geoff Dyer for non-fiction and a special mention for What I Talk About When I Talk About Running by Haruki Murakami if you do any running. Here’s the full list, this time they’re in the order I read them.

Tokyo Year Zero (Tokyo Trilogy 1) by David Pearce
4*. Atmospheric crime thriller set in Japan immediately after the war ended.

Heat by George Monbiot
4*. Good overview of the issues and a set of good solutions. Still very worrying that we’re not doing anything about any of it.

The Blank Slate by Steven Pinker
4*. Always interesting

A Florentine Death by Michele Giuttari
3*. Crime novel set in Florence written by an Italian policeman.

Gemma Bovery by Posy Simmonds
4*. Funny and sad

The White Tiger by Aravind Adiga
4*. Good story, memorable character.

Psychogeography by Will Self
4*. Excellent compendium of his interesting and funny pyschogeography essays. I think I enjoyed this more than when I read the odd one in the paper.

Perdido Street Station by China Mieville
3* . Good story, I just don’t really like fantasy as a genre.

Musicophilia: Tales of Music and the Brain by Oliver Sacks
-. Didn’t finish, a bit disjointed.

Pattern Recognition by William Gibson
4*. Good to read on the flight between London and Tokyo.

Hokkaido Highway Blues: Hitchhiking Japan by Will Ferguson
2*. Easy read with lots of anecdotes about Japan, nothing more than that.

Cities by John Reader
4*. Really good book about cities, where they came from and what they do. From the table of contents it looked like it might be a linear history but much more interesting than that. “The human gait is inefficient, on a par with penguins.”

The Casebook of Victor Frankenstein by Peter Ackroyd
2*. Dunno, just not the best book to read in the hottest week of the year.

The Ascent of Money by Niall Ferguson
4*. OK, I reckon if I read this again I’ll probably nearly understand the building blocks of the world’s financial system. I’m a bit financially illiterate and this made sense. “The concept of interest was probably derived from the natural increase of a herd of livestock”

Hertzian Tales by Anthony Dunne
3*. Interesting things but a bit too conceptual to be useful. “I use the term ‘genotype’ as an alternative to ‘prototype’ to shift importance away from whether or not a conceptual design technically works, to the ideas it represents”

Oryx and Crake by Margaret Atwood
4*. Post-apocalyptic tale of GM gone wrong.

The Craftsman by Richard Sennett
4*. Wide-ranging book on craft and craftsmen, it was an interesting read, possibly with a few too many confusing metaphors, but lots of fascinating anecdotes and useful thoughts. I blogged about what it has to say about innovation

A Certain Justice by P.D. James
3*. Standard PD James and Inspector Dalgliesh.

Batman: Year One by Frank Miller
3*. Seemed surprisingly close to the film, if I remember it correctly, though it concentrates on Lieutenant Gordon.

What I Talk About When I Talk About Running by Haruki Murakami
5*. Anecdotes about his life and what running means. Loved it.

V for Vendetta by Alan Moore.
3*. Again, seemed surprisingly close to the film.

Shapes: Nature’s patterns by Philip Ball.
3*. OK, not as inspiring as I hoped, and not enough detail for me to try simulating things.

The Age of Wonder by Richard Holmes
3*. It took me a while as it was a massive hardback that I didn’t want to carry back and forth in my bag everyday but an interesting journey through ballooning, astronomy and other 18th century science and the beginning of the Royal Society.

A Pale View of Hills by Kazuo Ishiguro
4*. Perfectly formed little book. How do you interpret the ending?

Status Anxiety by Alain de Botton.
4*. Brief look at how people and societies have dealt with status.

Anathem by Neal Stephenson
5*. Immense story of an alternative universe featuring 1000-year old philosophers.

The Ongoing Moment by Geoff Dyer.
5*. I know nothing about American photographers and this was fascinating. Loved his way of drawing connections between photos and photographers.

Wildwood: A Journey Through Trees by Roger Deakin.
4*. Made me want to visit forests. And buy a tree identification guide.

Matter by Iain M. Banks
4*. Latest Culture novel, I read some a few years ago and enjoyed coming back.

Performing Rites: Evaluating Popular Music by Simon Frith.
3*. For work, I’m interested in how people describe their musical taste. Fairly readable for an academic book.

The Secret Life of Birds by Colin Tudge.
3*. Avian equivalent to Wildwood above. Now know lots of useless facts. For instance, squirrels rotate their rear ankles 180 degrees when climbing down trees.

I also quickly re-read The Plenitude by Rich Gold and blogged about what it has to say about innovation

2010 was brought in by What The Dog Saw by Malcolm Gladwell and on the stack for 2010 are The Patient by PD James , Nocturnes by Kazuo Ishiguro, Staring at the Sun by Julian Barnes and The Nature of Technology by W. Brian Arthur.

All reading was tracked by the marvellous bkkeepr and graphed by the velocity of reading, which tells me my last 20 books, with 6936 pages in total, were read in 215 days. There was an average of 346 pages per book. That’s about 32 pages per day (or 1.34 pages per hour).

Reviewing electricity usage in 2009

2010-01-09T17:04:00.008+00:00

Timelapse movie of electricity usage during one day (more detail)

I've also been collecting my electricity usage during 2009 using a Current Cost smart electricity meter. I don't have it connected up to a computer all the time as that would require me to leave a computer on, somewhat self-defeatingly, so I had to regularly grab the data every 30 days or so. There are a couple of graphs below and the data used for the graphs is available for download in the unlikely occurrence that you'd like to use it.

This is the year graphed as daily kWh (kilowatt hours) readings in grey with a moving average superimposed in green. The zeros in the daily graph are days when I don't have any reading.

And this is the total usage in kWh per month.

There are other graphs I've made here, more information on how to get data out of a Current Cost in this post and more about our typical electricity usage here. Data gathered using one of these...

Listening to the radio in 2009

2010-01-06T16:40:00.001+00:00

In a possibly foolish feat of dedication I spent 2009 adding (almost) all of my radio listening to the BBC Radio Labs prototype, Radio Pop. The principle use of Radio Pop is to track online listening, but as most of my listening is done on DAB radios and podcasts I used the manual update feature of Radio Pop every few days to regularly add my listening. You can browse my complete profile at http://www.radiopop.co.uk/users/tristanf. As you can see I mostly listen to BBC Radio 4.

As 2009 is now over I've extracted all this data using the Radio Pop API, some Python scripts and a spreadsheet and this is what I found.

In total I listened to 1021 individual programmes for 633 hours and 17 minutes.

I listened to Radio 4 for 35,140 minutes over 974 individual programmes
I listened to Five Live for 1860 minutes over 31 programmes
I listened to Radio 1 for 417 minutes over 4 programmes
I listened to Radio 3 for 326 minutes over 5 programmes
I listened to Five Live Sports Extra for 228 minutes over 5 programmes
I listened to 6 Music for 24 minutes over 1 programme
I listened to the World Service for 2 minutes over 1 programme

(35,140 minutes is around 586 hours or an entire 24 and a half days listening to Radio 4.)

My top 20 most listened to programmes for 2009 were...

1. Today for 8712 minutes over 215 episodes
2. Saturday Live for 1795 minutes over 32 episodes
3. Any Questions? for 1479 minutes over 30 episodes
4. The Archers Omnibus for 1465 minutes over 20 episodes
5. In Our Time for 1335 minutes over 31 episodes
6. Woman's Hour for 1262 minutes over 22 episodes
7. Afternoon Play for 1041 minutes over 23 episodes
8. Saturday Play for 990 minutes over 15 episodes
9. Broadcasting House for 984 minutes over 18 episodes
10. Kermode and Mayo's Film Review for 960 minutes over 16 episodes
11. Pick of the Week for 927 minutes over 22 episodes
12. Simon Mayo for 900 minutes over 15 episodes
13. Desert Island Discs for 855 minutes over 19 episodes
14. The Archers for 705 minutes over 47 episodes
15. The Now Show for 636 minutes over 22 episodes
16. Front Row for 595 minutes over 22 episodes
17. The News Quiz for 549 minutes over 19 episodes
18. Classic Serial for 540 minutes over 9 episodes
19. Excess Baggage for 420 minutes over 14 episodes
20. Gilles Peterson for 417 minutes over 4 episodes

And then splitting all the listening data per day over the entire year gives the graph at the top of this post. The big gaps every now and then are mainly holidays. The regular cycle of spikes are weekends, when I tend to listen most.

It doesn't quite represent all my listening as I wasn't religious over doing this every day. It certainly misses out some podcasts which I listened to well after the broadcast date (mainly the excellent Thinking Allowed) and it's all BBC, but that's the only radio I listen to anyway. Whilst I was writing this up I discovered Dale Lane's excellent statistics of his TV viewing, which looks much more automated than my data gathering. But ideally we want this automatically aggregated and collated for us, just BBC TV and radio would be good enough for me at first. Obviously internet-connected radios and TV, from Olinda to the Sensia and iPlayer to Project Canvas, have the potential to do this but we'll need large-scale open systems from projects like RadioDNS, NoTube and URIPlay to get us to that personal media informatics utopia.

Mooso and moving forwards

2009-12-04T14:52:00.006+00:00

We’ve just launched the latest Radio Labs prototype, Mooso. It’s a game to play while listening to BBC 6 Music radio, enter tags to describe the current song and score points if you match other players. Sign up and play here or read more about Mooso on the Radio Labs blog.

But this is the end of an era, or something like that, as Chris and I have left BBC Audio & Music and are now part of the Prototyping team at BBC R&D where I'm the lead producer. Soon after we formed the A&M R&D team we held a planning week and scoped out four major project areas for us to tackle. One was around presence, attention and socialness for radio - this became Radio Pop and Olinda, another was around live events which became an internal-only prototype of a social gig attendance site, the third explored how the web could create new forms of radio content which evolved into the work we did exploring chopping up the Archers narrative and the final area aimed to develop new ways of music discovery. This final brief, combined with our interest in games, became Mooso. It’s certainly satisfying to have completed these four major prototypes, as well as lots of other things along the way including (with the help of James) setting up the Radio Labs blog.

These have been good times but the new team is just as awesome and I’m already knee-deep in many exciting projects, from representing narratives on the web to a number of possible prototypes around digital music. And hopefully, when I get some time, I’ll be writing about them either here, on Radio Labs or on the new R&D blog.

Some connections between (digital) books and music

2009-10-29T15:20:00.004+00:00

Just dumping some thoughts and links and connections about books and music in the digital world that I’ve noticed recently.

With the recent UK launch of the Kindle the news outlets were writing and talking a lot about digital books including the usual worries about piracy, DRM and price points (at the moment they seem to be priced similarly to physical books despite no printing or distribution costs). Which is the same set of things that the music industry went through years ago and which has now just about settled down, generally in favour of the consumer.

And I suspect the publishing industry will need to address some of the other problems we’re seeing in digital music - things like unique identifiers, search, content resolution and distribution. Which leads me to some things that all seem to link together somehow.

XSPF is a standard format for describing music playlists but Michael tweeted about the possibility of using it for lists of books - ‘xspf spec “An XSPF playlist describes a sequence of objects to be rendered. Objects might be audio, video, text, playlists or any other media type” erm, book lists?’

Playdar is a music content resolver which has been getting a lot of coverage recently - give it a song name and it will search for it on your hard disk, across all your computers, on your intranet and across music services on the internet. The latest README on Playdar says…

"Depending on the plugins loaded, Playdar could be used to resolve anything.
Here's a fictional example query for an academic essay:
{
   "qid"    : "XXX789",
   "author" : "Jonathan swift",
   "year"   : 1729,
   "title"  : "a modest propsal"
}
... and if you have a resolver plugin that could search for essays and academic
papers, it might respond like this:
{
   "sid"    : "YYY123",
   "qid"    : "XXX789",
   "result" : {
       "author" : "Jonathan Swift",
       "year"   : 1729,
       "title"  : "A Modest Proposal",
       "score"  : 1.00,
       "category"   : "humour",
       "url"    : "http://art-bin.com/art/omodest.html"
   }
}
In this example, the resolver matched the query to an essay posted on the web
instead of the local filesystem."

Could we have content resolution for books, essays and academic papers?

And finally, maybe bringing some of these together, there’s the recently announced Internet Archive’s BookServer which is, as I understand it, a federated search architecture for discovering, buying or borrowing digital books. There’s a little more explanation of it in this presentation but not a huge amount of detail yet.

Rural computing and nature's delights

2009-09-23T17:00:00.001+01:00

Russell Davies has written a somewhat contrary piece on the urban computing movement and addresses what could be called "rural computing". For that he gets my support and some thoughts.

"I suspect one of the reasons the countryside gets overlooked in all these conversations is that the aesthetics are so disappointing. Certainly the natural stuff's good; landscapes, hedges, skies etc. But as soon as something gets designed it looks like either Poundbury or Hobbiton..."

True, but I think nature's aesthetics are more than just good - the design and engineering of nature is marvellous - and designers, engineers and architects are already using biomimicry and copying the best bits.

And the thing that always gets me is that the countryside is full of delights and beauty; the heron guarding his rapidly evaporating pond in the late summer, rivers glinting in the evening sun, autumn leaves on the ground, unexpected birdsong and glimpses of deer in the morning mist. I think I want a kind of slower, maybe seasonal, noticings for the countryside...

The Plenitude and innovation

2009-09-12T17:49:00.004+01:00

I’ve just quickly re-read The Plenitude by Rich Gold while researching something. It’s a lovely little book about the “…dense, knotted ecology of human-made stuff ‘the Plenitude’”. But for now here are some dog-eared passages I picked out about creativity and innovation.

On creativity…
p.5 “In this context, creativity is not just making things (factories do that), it’s creating new things, things that have never existed before.” and also p.54 “…I don’t mean that stuff has to be physical. It could be an idea, or a concept, or a string of words.”

On problem solving, engineering and design (try to ignore the “hats”)…
p.25 “The hat of engineering is close related to the hat of design. Both work from need and desire. Both are concerned primarily with the user and the world - the “real world”, as they like to say. Unfortunately, in most companies design is pitted against engineering, a battle that tends to reduce the effectiveness of both. I think this is caused by a misunderstanding by both engineers and management, who see the hat of design as the hat of art. They think that designers work from inner vision and not problem solving.”

Chapter 3, on the seven patterns of innovation…

“Necessity is the mother of invention. Find a problem and solve it.”

“It’s a thing of genius. I had a vision and just had to do it.”

“The Big Kahuna. Scientific deduction of stuff from 1st principles.”

“The future exists. We just have to intersect it at the perfect moment.”

“Colonization. Find the unowned; package it; sell it back.”

“Stuff desires to be better stuff. Humans are how stuff makes more stuff.”

“Change the definition. Language and metaphor create/are the world.”

Which includes this on corporate R&D…
p.38 “A company can build too much of an ivory tower. While visions don’t flow directly from problems, they often grow, like a pearl, from an irritating grain of sand.”

On naming and defining things…
p.48 “The seventh pattern, Change the Definition, is the one that designers often invoke when they work on corporate identity construction […] These changes allow their respective companies to innovate in new areas, to develop and create in new ways, to behave differently. In a very real sense it changes the frame of imaginable invention.” and p.49 “The definition of a product (the product genre) determines to a large degree what it can and cannot be. It sets up the frame of expectations, not just for the customer but for the producer as well.”

On how we need to innovate…
p.55 “And since we can’t make what others are making - by law and by the law of the marketplace - it is only through creativity and innovation that we survive.”

And I’ll finish off with a couple of passages that are not so much on these themes, but that I found liked.

On brands and nature…
p.63 “There are differences between the bio-Plenitude and the stuff-Plenitude. As far as I know there is nothing quite like brands in nature.”

And on the fractal nature of stuff…
p.67 “I once asked Hal Varian, the economist, how much stuff is there in one room? “Well, what’s the definition of a piece of stuff?,” he asked. “How about: something that was individually designed, shipped, marketed, and sold,” I replied. “OK,” he said, “then there is no answer, for stuff is fractal.”“

On a final note, there’s a companion autobiographical PDF which is well worth reading alongside the book.

The good craftsman and innovation

2009-08-05T19:51:00.003+01:00

The Craftsman by Richard Sennett

I'm just finishing this book on craft and craftsmen by Richard Sennett. It was an interesting read, possibly with a few too many confusing metaphors, but lots of fascinating anecdotes and useful thoughts. So I'm going to blog a dog eared page. Around p.262 the author writes about the 20th century philosopher Wittgenstein and his obsession with perfecting the architecture of his sister's house. At one point he had just about finished it all when he decided the ceiling of the drawing room needed to be raised by an inch. Anyway Sennett uses this as a jumping off point to write about how to manage obsession.

Paraphrased...
The good craftsman understands the importance of the sketch - that is, not knowing quite what you are about when you begin.

The good craftsman places positive value on contingency and contraint.

The good craftsman needs to avoid pursuing a problem relentlessly to the point that it becomes perfectly self-contained.

The good craftsman avoids perfectionism that can degrade into a self-conscious demonstration - at this point the maker is bent on showing more what he or she can do than what the object does.

The good craftsman learns when it is time to stop. Further work is likely to degrade.

I liked that. And those five points certainly apply to our prototyping and innovation work on the web at the BBC, indeed I've got some more notes that I need to write up about this sometime. But is what I do a craft? I don't know. And anyway what is it that I do exactly with the internet and technology? Produce, manage, build, innovate, design, code, craft?

Instapaper and reading on the iPhone

2009-07-30T17:59:00.004+01:00

I've been finding Instapaper really useful lately for reading longer articles from the web.

Instapaper facilitates easy reading of long text content.
We discover web content throughout the day, and sometimes, we don’t have time to read long articles right when we find them.
Instapaper allows you to easily save them for later, when you do have time, so you don’t just forget about them or skim through them.

A one-click bookmarklet adds it to your list and there's an iPhone application for reading the articles. It works for me, giving good clean text for in-depth reading on my iPhone during my daily train commute. It's more like reading a book than a feedreader and I think that mode of reading is valuable. Anyway, some of my recent favourites...

Paul Morley on how a year studying at the Royal Academy of Music changed him
Leviathan or, The Whale | PD Smith
Pitchfork: Poptimist: Poptimist #23 on the charts
3quarksdaily on "DESIRE PATHS: READING, MEMORY AND INSCRIPTION"
The Best Bits » American Scientist - A new technology called compressive sensing slims down data at the source

If you use Instapaper you can subscribe to my starred articles using my username, tristanf.