The main reason I’m keen to get him interested in this is that I think that we are increasingly being divided into a role of producers and consumers of digital services. It’s about empowerment.

I’d argue that the definition of ‘empowerment’ is laden with unspoken social-class bugs (or features, depending on your social class viewpoint).

To my mind, this is a profoundly political issue, but I’ll let Eben Upton and Theo Blackwell explain why this matters much more efficiently than I can here.

Yesterday morning, I was a bit worried. I wanted to take my son to a meetup where he can be enthused about the possibilities offered by his Raspberry Pi.

I want to strike while the iron is hot after he’s opened his presents. I also like the idea of involving young people in this kind of thing. Earlier this year, I organised an Open Data Day for school pupils in the London Borough that I live in. I wanted to encourage them to get under the bonnet of government data. More about that here if you’re interested.

I asked, on twitter, if anyone knew of any #Raspberryjam meetups soon after Xmas in or around London. There aren’t weren’t any planned, but my Twitter feed convened one in no time at all.

John Bevan of Mozilla (the people behind Firefox) ponied up an offer of a central London venue for a meeting on Thursday 3rd January, and Rob Bishop of Raspberry Pi quickly followed with an offer to attend and help out. A few others with experience in running ‘Raspberry Jam’ events also volunteered to help out, and …. bingo! We’ve got a plan.

So: Book your tickets now! And please please please only book a ticket if you’re going to come. And if you change your mind after booking, please take the time to log back into Eventbrite and give your ticket back.

Update: I’ve decided to charge £5 to people who book but don’t turn up. You pay £5 to get the ticket and we give you a £5 note when you arrive. Unfortunately there’s a booking fee that we can’t refund but it’s <£1.

I suspect demand will massively outstrip supply, so please bear this in mind.

Crowdsourcing analysis for policymakers

How open data is being used government, how it could be used as a participative tool, and what the opportunities / pitfalls could be.

Presented by Andrew Stott

Co-design and policymaking

A look at the practicalities of involving large numbers of people in planning and designing policies, followed by a discussion of the politics and the ethics of ‘collaborative authoring’

Presented by Steph Gray

Policymaking in the cloud

A discussion of new ways of doing things that arise from more dispersed technical networks. ‘Scrum’ project management, open source development and, Peer-to-Peer organisation have all been held up as being ideas that politicians and governments can learn from.

Presented by Dr Andy Williamson

Quicker, cheaper and easier than polls

In the past, opinion polls and focus groups have had a great deal of influence over policymakers. Today, the social media ‘firehose’ provides us with a torrent of opinion and sentiment to draw from. How is this done? And how well does this commercial practice apply to policymaking?

Presented by Dr Nick Buckley

What policymakers can learn from gaming

Technological entrepreneurs have become adept at finding new ways of motivating people, not only managing to change their behaviours but also encouraging them to develop skills, mentor others and solve problems. What can politicians learn from the most successful interactive-content industry the planet has known?

Presented by Jude Ower

Are we too irrational to participate in policymaking?

Can a government that adopts approaches from behavioural economics be trusted to be serious about any kind of participative politics? Are the two oil-and-water opposites? Or is it more complicated than that?

Presented by Warren Hatter

In other developments, we have been working on an ‘open data for schools’ project, and the Political Innovation approach to understanding policymaking in the digital age has formed the basis of a number of one-day professional training events.

I won’t go into too much detail about his now as I’ve covered it extensively on another blog with this list of posts. In summary, though, the educational possibilities are very interesting, but I would suggest that developing this idea is an almost essential step for any polity that wishes to engage large numbers of people in rational policy-making processes.

What’s the next step? Well, I’ve pulled together a draft report here, based upon experiences gathered organising a Schools Data Day in association with The London Borough of Barnet and Deloitte LLP, and there are a range of pupil presentations that came out of the hack day that I’m planning to show in some presentations shortly.

I’m hoping to be able to gather enough interest to do a bigger one of these shortly.

Firstly, I completely buy the argument that it advances: That the notion of the ‘social graph’ is a very limited one and it is stuck at the limit to which we are all machine-readable as individuals. On the other hand, I felt compelled to read it because a few influential people within my social graph said it was worthwhile.

Yet, for all of the shortcomings of the concept of ‘social graph’ as a means of mapping general relationships, particularly for marketing purposes…

“Google, for example, uses XFN as part of their Social Graph API. This defines a set of about twenty allowed relationships. (Facebook has a much more austere set: close_friends, acquaintances, restricted, and the weaselly user_created).

But these common relationships turn out to be kind of slippery. To use XFN as my example, how do I decide if my cubicle mate is a friend, acquaintance or just a contact? And if I call him my friend, should I interpret that in the northern California sense, or in in some kind of universal sense of friendship?”

… it’s still a useful heuristic for the narrower purposes of understanding personal influence. We need to first understand what it is to understand what social media tools are trying to achieve. They are trying to find ways in which they can make our relationships more machine readable. There is a lot of effort coming from social media platforms at Google+ and Facebook with a view to monetising our social graph.

So the Pinboard blog has identified a large hole in the strategies of Facebook and Google+. But unless you’re a shareholder, why should we bother about this?

I’d say we shouldn’t. What is of interest, though, is how we can use the tools that we’re getting for free (!) to achieve things.

In my line of work, the big question is how we understand (and exercise) influence. I think we can learn something about this from the concept of the social graph. Remember, at the start of this post, I said why I’d read that Pinboard article? Four people who influence me all linked to it. I think it was four. I can only name two now. But I’d noted the link, at that’s the important thing. It may even be the case that one of the people who linked to it is someone for whom I’d generally ignore their material. But that doesn’t matter.

I’m pretty sure that politicians and journalists respond to ideas and concepts in the same way.

So how does the concept of the social graph help here? For example, this is an old-ish app Facebook: The Friend Wheel. I’ve highlighted friend Dominic Campbell at random here, simply to illustrate that he knows a lot of people that I know (52) pm Facebook.

My personal friend-wheel. Click to enlarge.

This shows all of my Facebook friends and who is connected to who. On the top left-ish end, you’ll see people who are densely connected to each other – often with me as the connector. On the bottom right-ish, a lot of people who barely know anyone else from my circle.

Then I pulled up another old Facebook app – this time, Friend Sets. Using this, I picked five friends who I’ve got to know at different times of my life to see how they connect to each other:

I’ve picked these names at random, and I’m using a free app here. With a better, three-dimensional one, I’d be able to identify all of the different social circles I’m in. Personally, I’m connected to circles that broadly consist of….

• family and their friends
• old school / college friends, childhood friends, etc
• people who I live/ have lived near, or shared a flat with
• people I used to work with in different jobs & some of their friends
• fellow UK Labour Party supporters
• people I know from a village in Ireland that I visit regularly
• people who are in my political ‘cell’ (humanist, internationalist, democratic-lefty, pro-EU) and their friends
• people I’ve met in Northern Ireland’s political circles (I’ve taken an interest over recent years)
• people who are active thinkers around social media and local/central government
• people who were active in UK political blogging (2005-9 mainly)
• people with a general interest in new media and politics
• a very small handful of random unconnected people who don’t fit anywhere else (holidays etc)

The venn-diagram of that lot has a few crossovers. There are one or two interesting surprises in this (“How come you two know each other??”) LinkedIn is often even more surprising in this respect, but less generally interesting. And within those crossovers, there are often people who are well-respected in more than one sphere. . But as the Pinboard blog article says, this intelligence tells us very little that is useful about the thousands of two-way relationships that this represents.

I mention all of this as a prelude to a question I’m going to write about shortly: People may not have a useful social graph, but is there such a thing as a conversational graph?

I think that there is, and that it is a very valuable thing if we can identify it. I’ll be back with a posting on this shortly.

For me, the most interesting aspect of this is that it gives journalists, PRs, or even more sophisticated intelligence gatherers, the chance to see things through the filter of the Internet’s hive mind. Journalists, PRs and researchers can learn from Hedge Funds and doctors here:

The interesting information can be found in changes and in the related stories that are spinning around a term that you are watching. A significant shift in the computer-tracked sentiment is the thing that tells the story. Hedge Funds have jumped on Twitter precisely because of this. So have doctors looking for ‘flu epidemics.

In the wild, predators notice movement. The market of human interest is the same. We process newness and change.

To understand a developing situation, the firehose (with the right analysis) can tell us when a new dimension to a story emerges or attitudes to an existing aspect of the story changes noticeably. As predators know, we need to be looking at the space where change is happening. Large numbers are often less important that sharp variations.

Case study: Noticing that a story is happening and what the key factors are

I’d now like to start to look at how we can dig into a story – in stages. I’m doing some work with some developers on a tool called Repknight, and they are aiming to create one-click ways of doing a lot of what follows, but this post is intended as a bit of a slo-mo walkthrough what is possible.

Once we have all of that data in one place (providing we have the processing power – i.e loads of servers), we can start to run all kinds of analysis over what we have. Sentiment analysis is one of the most common processes that we can use to sift this data.

Take a search term. For illustration purposes, I’ll use the recent ‘Dale Farm’ evictions as an example of a developing story. The term ‘dalefarm’ was a text-string that was being used by critics and supporters in mentions of the evictions on social media outlets.

Mining online comments with ‘dalefarm’ in them was, therefore, a useful way of keeping tabs on a developing situation.

So what could we find out?

Let’s start at the most obvious level: Can we can tell if something is happening at all? Here is a graph from Repknight showing mentions of this term across a wide range of social media platforms (Facebook, blogs, Twitter, YouTube, Flickr etc)

Yes. We can safely say that on the 18th and 19th October, something was definately happening. While Dale Farm may not be the best example of this (it was a ubiquitous news story that we all knew was going to happen), if you are monitoring a particular term (your organisation’s brand or name, a particular news issue, etc), this can be useful. Journalists, in particular, are looking for relatively large prolonged jumps (a ‘spike’ can often be a story that doesn’t have legs - a rumour that is quickly scotched).

Next, we need to drill into what was happening on those days during a short time-period around this term. We can look at a list of other words that also appear in comments with ‘dalefarm’ in them – like this one (click to enlarge).

Here we see a map of the protagonists around this story – the police, Basildon Council, Richard Howitt (an MEP who got involved), bailiffs and, bizarrely, Newts. Each player in the story has some positives and negative sentimented comments around the story on a range of different media platforms.

So, what use is this to us? Sentiment Analysis in itself, isn’t hugely accurate as a quick play with this entry-level tool will show you. As a way of understanding individual short messages, it is deeply flawed.

But automated sentiment analysis is getting better (and people are often bad at detecting sentiment accurately as well!). Changes here are what matters – as investment analysts and epidemic-tracking medics will tell you.

I noticed towards the end of last week that the sentiment around the ‘Occupy London Stock Exchange’ story (#OccupyLSX) was going from a fairly positive general response to quite a negative one. I’d already registered that there was a developing fuss around access to the St Paul’s but I’d tuned it out, thinking it was one of those aspects of the stories that reporters were talking about to fill in time.

But drilling in to that day’s sentiment, I found that ‘St Pauls’ had a strongly negative balance. This wasn’t going to go away. Throughout the day, this grew as the closure of the Cathedral was picked up by opponents as well as neutrals, it became the aspect of the story that dominated the news bulletins.

This information is only one of the building blocks needed to cover a story. All of the information I’ve mentioned here so far is available within a couple of clicks. Repknight are allowing users to build up a store of data around particular subjects over a long period of time, thereby allowing users to be able to identify significant changes properly.

In a subsequent post, I’m going to look at how we can dig into the negative messages, identify the communities that are talking among themselves, identify the influencers within those communities and the connectors between them. We can see who is making the running on a particular story – and even intervene with them to influence how a story unfolds.

The Firehose. Fancy a drink?

Today, I’d like to focus on how we can start to intelligently mine that huge firehose of information and extract useful, meaningful information from it.

This is particularly exciting for us because it’s information that we’ve never been able to see before – a huge torrent of observation, comment and hard data that is being produced in a machine-readable format for the first time ever.

To put this into context, take any point in recent history and imagine you could get millions of people to come to you and volunteer opinions or interesting extracts from the things they’re read or seen in a way that we could tabulate it, weigh it for value and get information from it.

On a slightly darker note, imagine that they do this in a fairly guileless way – often even putting info into our hands that we could exploit or abuse.

Imagine what the Stasi could have done with it? Imagine what benign forces of law-and-order could do with it in order to undermine crime or terrorism? Pop your ‘Liberal’ thinking cap and and spot the potential dangers here. Then try on your ‘sales and marketing’ cap and look at the opportunities or your ‘consumerist’ eyes to see the threats here.

It’s all exciting, fascinating and worrying in equal measure.

So what can be achieved – with the right tools?

Think about Twitter, for example, as a huge database of these comments.

That’s basically what it is, after all. 200,000,000 comments a day. An utterly vast database that constantly imports a torrent of characters.

Let’s pick a random tweet for illustration purposes.

RT @andrewtghill: An old FT column in which I tackled Adair Turner's provocative 2009 airing of the Tobin tax idea: http://t.co/nVii9rOU

September 28, 2011 11:28 am via TweetDeckReplyRetweetFavorite

@umairh

umair haque

Here we can see the following things:

• A sentence that has meaning of some sort
• A collection of words that we can juxtapose (FT, airing, tax, idea, etc)
• A tweeter about whom we can draw some conclusions (how influential they are, who they associate with, sometimes, where they live and work). There are lots of clever tools that help to determine & qualify ‘influence’ and Umair’s Klout score is very impressive here
• We can sometimes aggregate their comments to draw conclusions about what they do for a living, what their interests are, etc (again, Klout tells us Umair is influential about capitalism, politics & markets, but there’s plenty more that we could find out from his Twitter profile and what he says)
• In a small percentage of cases, people even have ‘geo-tagging’ enabled on their tweets so we can nail down whereabouts they’re tweeting from – so we can run a search on people in a specific area if we like (usually at 25m radius)
• If not, there are plenty of other clues – his profile says London/West Coast and conclusions can be drawn from his subject matter and who he associates with
• They may have included links to web-pages or pictures – sometimes the web-page URL may include a particular word (eg www.guardian.co.uk/football/2011/sep/28/carlos-tevez-denies-refusing-play) – and those words will be added to the list of words that can be juxtaposed in that tweet. This also applies to urls that have been masked by an ‘url-shortener’ such as bit.ly or Twitter’s own shortener.

The more slowly and painstakingly we look at a collection of tweets, using human eyes, the more we can find.

But that’s a huge job – and often human eyes often bring unconcoious biases, selectivity and mis-perceptions to the table. And as any marketeer or pollster will tell you, the real value is often in looking at what comes out when we sift large volumes of data rather then small snapshots.

So we may want to get cleverer about the way our machines process this information and extract data from it.

Now imagine you have a vast bank of computing power at your disposal. Imagine twitter have let you access this huge database/torrent of information in a way that allows you to process it cleverly.

At the crudest level, we can run a sentiment analysis programme over a particular tweet. I’ll be posting on sentiment analysis and other forms of automated language processing in more detail shortly, but please bear with me for now and accept my assertion that it can machine-read large amounts of information and tell us whether references to (picking a name out of the air) Adair Turner are generally positive or not.

We can run a query to see what other words crop up in tweets that also include the word Adair. Of course, there are other people called Adair around and our sentiment analysis will never be perfect, but we can look at the context of each of these words (i.e. words that occur on twitter and also occur in tweets with the word ‘Adair’ in them) and draw conclusions about them as well.

Do those words occur in a context that lends positive, negative or neutral sentiment towards them?

We can look at a graph showing when these tweets happened. Are there real-world events (TV appearances, news stories, etc) that provoked them?

Then lets take that single-line graph showing the numbers on a timeline and give it a colour. Let’s say green = positive, red = negative and blue = neutral. If a particular real-world events results in that graph surging and changing colour, we can learn something from that.

Has our subject said something interesting? Has the hive-mind of the internet seen it as being significant? If so, it will rise to a peak and decline slowly as people keep talking about it. Or was it a claim about them that gained little credibility – one that spiked and dropped quickly.

For a politician, marketeer or a journalist, these changes provide valuable communications information. Hedge funds have found this information more valuable than other available data. Health authorities find the information more useful than other available data in tracking epidemics.

As a campaigner or marketeer, you can see if a damaging line of attack on the way earlier. Has something been said that needs to be clarified? Is there misunderstanding or mischief coursing around?

There’s also the opportunity that this kind of insight provides to anyone who wants to intervene and change the way the firehose is talking about something.

Can we drill down into this information and find out who the influencers are behind a surge like this? Can we contact them directly with the clarification or rebuttal? Or can we give rival influencers the ammunition they need to launch a counter-attack?

Journalists watch this firehose. I alerts them to stories. It gives an good indicator to the credibilty of a story. A rapid rebuttal can take a problem off the radar.

Over the next few weeks, I’ll start posting some graphics here with a few example illustrations. Please stay tuned!

Napster: It dragged music into the machine.

OCR – Optical Character Recognition - can take old-media (in this case, print) and turn it into digital media. I can scan that pamphlet that I wrote in the mid-1990s and start cutting-and-pasting bits onto various sites I manage, or emailing bits to people who should have read it at the time, dammit!

We’re gradually making print ‘machine readable’ in this way, like Project Guttenberg or Google Books has done. And before the music-on-demand service Spotify came along, Napster galvanised legions of music-pirates to get music digitised and available – by encouraging users to rip their CD collection and share it.

Call Shazam from your phone and it will match music you’re listening to with it’s digitised database. It will text you the song title. Nearly ten years on, this is still the most startlingly cool thing I’ve seen digital media do.

I work with a range of organisations that use social media to build relationships with the various circles they (want to) move in.

Social Media training - about more than mechanics. Thanks to @adrianshort for this one - click pic for credit

It’s easy to fall into the mechanics of it all – a briefing on how to perform the various chores that social media’s more adept users perform or a bit of analysis showing what makes for good viral content.

It’s also tempting to wheel out a few of the mind-blowing statistics that occasionally scare even the evangelists. Both of these activities have their place.

But I’ve found that it’s often better to ask people why all of this is happening. OK, there are technological reasons: Computing power is getting cheaper.

Networked devices have gone through design and usability revolutions that mean that they integrate with our life more seamlessly. For a long time now, our access to the network isn’t just based on the desktop PC. There are 31 Billion devices and 4 billion people forecast to be connect to the internet by 2020.

At this point, to help move things on a bit, I like to retreat to a little bit of simplification.

Let’s think of the Internet as a big conspiracy. It wants us do do everything online so that all of the data can be processed. The more it knows about what we do, and when we do it, the more efficient it can become at meeting our needs.

It wants to know when we wonder about anything (Google), buy anything (Amazon), go anywhere (Foursquare), associate with anyone (Facebook), phone anyone (Skype), or make ourselves a cup of tea (Twitter). Where older functions are broken (e.g. spam/e-mail), new modes step in to provide a fix. Twitter, Blogger and a range of other tools co-opt us into indexing the Internet for Google (at no charge).

We’re heading towards a point where those 31 million devices provide the network with a pretty detailed portrayal of what we do all day long. Colbert said that…

“The art of taxation consists in so plucking the goose as to obtain the largest possible amount of feathers with the smallest possible amount of hissing”

Social media is about getting us to breach our own privacy without much hissing.

You can see this as a sinister capitalist, consumerist or big-government plot. Or it could be a liberating phenomenon that personalises, streamlines, and liberates us from chores and choices.

Whatever. Either way, it’s winning. Annoyingly for people who don’t like it, boycotting it is about as effective a form of resistance as pacifism was in 1939.

So, if you want to work with it, you have to become machine readable. You have to be easy to contact. You have to compete for the attention of millions of people who no longer focus on a handful of channels. That means getting into their peripheral vision wherever possible.

This could mean doing clever things with open data – something that I reckon every communicator needs to be thinking about.

But let’s start from the beginning: Whenever we use social media tools, we join ‘the firehose’ output. People can analyse it to find out what we know, what we think or who we’re influencing.

The machine is reading us. We have to understand that – when we use social media – we are preparing ourselves to be machine-read. It means that – if we put content out there – we have to understand how others will read it. What they’ll read it on, how it’ll look to the search-engine link that leads us to it.

We can control all of that. Look at the Google results that lead visitors to your page to see how well you’re doing that. Simply taking two examples at random:

1. My local bike shop, Shorter Rochford Cycles (as Google sees it):

 2. The big-brand cycle retailer, Halfords (as Google sees it):

See the difference? Halfords have used their bigger budget to hire a site-designer who has made the site more machine readable.

The code tells Google what the site’s structure is so visitors can jump directly into the page they want to, and there’s just the right description under the link.

Whenever we use social media, we’re competing with people who are also playing this game. By understanding the different ways that machines read what we do, we play it better.

Does this all sound complicated? Remember, social media is part of this great conspiracy to get us all to be machine readable. Using social media tools properly can usually enable most of us to do this really well – without ever reading a book about Search Engine Optimisation.

That’s why it’s worth thinking a little bit about how you use simple things like Twitter and Facebook.

Twitter: We only see the tip of the iceberg

This changes the very nature of decision-making – and by implication, of government and governance. Governments can even foresee revolutions if they know where to look for them.

I spend a lot of time training clients in the use of social media channels and how they can be used to communicate – in the fullest sense of the word. One of the earliest points that I make to users who have a reasonable amount of experience with Facebook or Twitter, for example, is that these media are not as friendly as they may look.

Sure – most of us have a mostly-friendly experience of them because we view them selectively. By definition, we only see the contribution to Facebook by our friends. And there is at least some level of affinity with the people we follow on Twitter. We could make a decision to only actively follow White Supremacists if I wanted to see the other side, but even Glenn Beck has found that to be a career-limiting activity.

Social media: Not always a friendly mirror (click pic for credit)

So. We only see a fairly friendly social media, in the same way that we usually (!) see a friendly face in the mirror.

Even then, we still only see the tip of the iceberg: Most users only have a limited view of Twitter. Twitter imposes limits on what we can search and we only see a fraction of the results that, in an ideal world, we could see.

Unless you have access to the Twitter ‘firehose’.

And unless you have a huge amount of processing power to examine those results.

And unless you have the knowledge and tools to know what you’re looking for, how to find it, and how to extract useful knowledge, or ‘alerts’ from the data.

Here’s why: The most recent statistics that I’ve seen tell me that there are 230,000.000 tweets sent every day. That’s about 160,000 per minute or 2,662 per second.

See? This is why it’s called a ‘Firehose’

So using Twitter to detect Flu outbreaks, for instance, isn’t as far-fetched an idea as it seems. Using it as a customer service tool is a no-brainer.

I’ve been given access to that ‘Firehose’ along with the kind of analytical tools needed to make sense of the information that it lets me see. I’d be interested to talk to people in both government and campaigning roles on how this can be used more effectively.

No-one explained the value-model as well as Dan at ALittleBitOfSomething dot com.

Here’s a good post with another load of them – including this one (perhaps the most mild mannered one of all of them, but my favourite):

An email with one of those images (and perhaps other messages beside them) is very likely to spread. If it’s a web-link, it can be an image with advertising placed beside it, thereby becoming a powerful bit of marketing.

Of course, this phenomenon has a darker side. The racist joke is generally a combination of a racial slur couched in a few sentances that some of the hearers will remember and relay to others. It’s likely to travel much faster than the unadorned message that “[insert chosen target group] all smell”

In the 1950s, the Situationist Guy Debord championed a form of artistic activism known as détournement – a form of artistic pranksterism in which capitalist imagery was appropriated, adapted and used against itself. The Adbusters site – one of the more established online art collectives – has some very good examples of images that have the politically subversive role of trashing brands. Take this example – digging at the way that fashion industries play in insecurities:

For me, it gets a bit too earnest and loses it’s impact when they feel the need to expand on the idea though:

Viral imagery potentially transforms communications. The right image can reach millions of eyeball without any production budget behind it. Brands can be trashed, opponents can be riduculed, and (often over-simplfied) ideas can be spread. It’s also a phenomenon that no serious campaigning user of social media can ignore.