About 2 months ago, I started to use Brandwatch’s service on my own. The company had made me an offer to become its COO and I wanted get to grips with the systems and test it out. Rather foolishly, it turns out, I did this without any of the usual training and before I knew about the fast and responsive tech support. And that is where I started getting lost: I could not for the life of me figure out the difference between a workspace and a query.

Too proud to actually admit my misunderstanding when I joined, I got the chance to see our chief report writer, Phil Newman, in action setting up a few queries and a workspace. He was carrying out some in-depth social media analysis for one of our clients and their competitors. It was great to see how he went about it as when running home the next day, I had something of a Eureka moment and the classroom/student analogy popped into my head.

I realised that I, as the Brandwatch user, could be thought of as the teacher, the workspace as my classroom and the queries are the students in the classroom. As the teacher (Brandwatch user) I decide which students (queries) I wish to compare and what I wish to compare them on. I can decide how many students to include in my classroom (workspace) and even whether I need 1 classroom or several.

Classroom with Three Figures by Lavern Kelley from Cliff1066TM

Let me try to illustrate my analogy with an example. Say I wish to buy a family car with three seats across the middle in both the front and back rows. My initial Google search has identified two good options; the Fiat Multipla or the Honda FR-V. So I set up the a query for each of these car models, wait 20 minutes for all the data to get populated and then open a workspace. As I want to compare these two models on a like-for-like basis I use the first few tabs of the workspace to compare the two models. I create a ‘new query group’ to do this and then click on the components called ‘History for query group’ and see how many more mentions one site gets vs the other. Adding a few more ‘query group’ components and I can compare the sentiment, see which vehicle is being mentioned more on the most important websites, etc. I may be stretching my analogy too far here, but each tab could be a different subject (French, English, Geography, etc) and each component a lesson. In the same workspace, I can also look into detail on either one of the models, for example looking at the topics that are most prevalent.

In the above example, the workspace was used to compare different queries as well as focus on one query, just like a classroom is used to teach different students and provide some one-on-one attention for any given individual.

Now my analogy is not perfect, but, for me, it was something of a Eureka moment. I apologise if this is obvious to you, but that is probably beause you have had the benefit of the training and the help of the tech support team. Or maybe I am just a bit thick; back to school for me perhaps?

Image by cliff1066™ licensed under Creative Commons.

So if you want to be REALLY up to date, come and give Brandwatch a go.

As developers, we may not be able to directly answer those questions because typically they’re subjective. But what we can do is provide meta-data that helps our users make that judgement. At the beginning of October, we added a new feature to Brandwatch: metrics about the source of each page. We’re using a couple of excellent services to bring in this meta-data: the mozRank API from SEOmoz and the Twitter API.

The mozRank API provides some very useful site metrics, including the number of backlinks to the site and the mozRank, which is a score from 0 to 10, giving a feel for how important the site is. I tend to think of the mozRank as being like Google’s PageRank – it tries to distill the number of inbound links (and how important those referring sites are, in turn) into a single score. For twitter pages, we now show the number of followers that tweeter has, how many tweets they’ve written, and the number of others they are following. You can sort the data using these metrics, to help sift out the most important pages from the passing chatter.

Choosing a username

was impossible. I wanted to link it up to my existing Google accounts, but guess what, they are all taken! So i had to settle for some random username which i will never remember and which pisses me off

Using Wave

Once through the username ordeal, the app loads nice and quickly and a bunch of stuff is staring me in the face. Here’s a picture

Dr Wave – aka – Greg the happy Aussie Wave product manager then gives me a 2.12 minute intro, care of an embedded YouToob video, which is cute and I’m off.

That’s where things get weird. I’m off, but I’m not sure where I’m off to. My contacts who also have Wave accounts show up, and I hook Wave into Twitter to get some content going, but other than that, it’s not at all clear to me what to do next. I start a conversation with Miles and see him typing on my screen as it were, which is nice, but not ground shaking.

And after 20 minutes or so I’m left with the following feelings about Wave

1. Nice UI, albeit quite similar in structure to an email client
2. Some nice Javascipt touches – like being able to select where in a thread you want to make your comment, just by hovering there and seeing a bubble show up and a great idea on scroll bars. This is a real issue for us at the moment with the new release of Brandwatch in testing – there are just too many scroll bars – Wave has a nice solution for this
3. What it really needs is ADOPTION and PLUGINS. I have a feeling that it has a chance to become something important, but it’s not clear what – at least not clear to me – at this stage. But when more people actually USE it and figure out it’s really good at this and that (and maybe not so good at other stuff), clarity will emerge. It’s an interesting tool, but what can be made with it is, I guess, up to us.
4. Final point is I send my congrats to the Wave team for coming up with something new and interesting and for Google to have the wherewithall to back them and get it out there. I’m sure it will spawn another round of web-based innovation and that’s good for us all

One of the difficult choices we face – one that I face from customer requests, our investors and my inner Magnus Magnusson, is where to best direct and deploy our fabulous, but finite development team? Do we put more emphasis on developing a user-friendly, fast, beautiful, full of wow-factor, no-training-required front end or a robust, scalable, fast, redundant, far-reaching backend?
The answer in the past has been, 8 out of 10 times, the backend. We are after all, a data company and it’s the backend that finds, organises and analyses the data.
But that has meant that our front end has been rather restrictive which can be a pain for our users as well as we ourselves using the system. No longer! We have pushed the weights along the scale a bit towards the frontend.

Not quite balanced as this cute little diagram suggests, but on the way.
So by the End of November – oh my – what UI delights will we have in store for the unknowing world? Some maybe!

signed: your rebalanced buddy

On the whole though for me the name Google is bathed in an overall sense of awe. When they launch a new product, they just do such a damn good job. And this is a company that is just over 10 years old. It’s not as if we are talking about decades of corporate learning here, unlike other super performing organisation like say Toyota or Apple.

So to Chrome – their new browser. Another amazing entrance from Google. Lovely and clean. It barely takes up any screen real estate, and there are some other nice design touches like the animations on file downloading and the incognito guy. BUT the best part about it for me and in particular for Brandwatch is it’s so goddamn fast. We did some side by side tests with IE7 and Firefox3 and Chrome is 50% faster at loading our app than either of the others. And 50 is a lot of %s.

The reason, it appears, is how Chrome deals with Javascript. We use a lot of Javascript in our UI to make it as nice to use as possible and firefox in particular is not that fast at rendering it. Chrome is. It’s my new browser of choice. Google has done it again – horray! boo! horray!!

One other way to look at this is: given some data, how do you turn it into useful information. And this is really, really subjective. What I consider as noise, you may consider as insightful – and vice versa.

Even assuming we agree on what we try to measure, and on what would make an interesting measurement, the interpretation of the information is by definition subjective. A good example of this is … the housing market. My specialist subject again.

Let’s summarise what seems to be the common understanding of the UK housing market – according to the press, tv, estate agents and other ‘experts’:

• the UK housing market has seen strong growth in the last 10 years
• this growth is supported by ‘sound fundamentals’ (high employment, housing shortage, low inflation and interest rates)
• if any slowdown had to happen, it should have been just that: a slowdown, a ‘soft landing’
• the US housing market ran into problems, caused by (and only by) subprime lending
• the subprime issue has forced UK banks to restrict lending, which is stopping people from buying houses
• there is no subprime in the UK

Bottom line: as soon as banks go back to normal lending practices, the market will boom again.

I’m not a financial advisor, nor am I an economics expert, but the above sometimes sounds like wishful thinking to me. It is an interpretation of some information, and it does sound plausible in some respect. But it is possible to come up with a contradictory interpretation which will sound just as plausible, if not more:

• the UK housing market has formed a bubble, with exponential growth, over the last 10 years
• sound fundamentals (assuming they are sound) can only go one way – they become unsound. It’s difficult, for example, to slash already-low interest rates. Some other fundamentals are just myths: how many empty houses can you have in a housing shortage?
• bubbles are not followed by soft landings
• lending practices were lax for all types of loans in the US, not just subprime
• UK banks now ‘restrict’ loans by requiring 10% deposits and only lending 4x salary. THIS is normal practice. If we have seen a fall of 70% in lending over the last year, it just means that 70% of all loans made a year ago were not viable. Here is the ‘UK subprime’
• people don’t want to buy houses anymore. Who would buy when the market is crashing by more than 10% annually?

Bottom line: the UK market is in for a serious crash, from which it won’t recover soon. The higher you go, the bigger the fall.

Endless blog posts could be written about the merits of either interpretation (and this is already quite long). But my point is: how do you get Brandwatch to build a sound, logical (irrefutable?) narrative from the data? That is one of the challenge we currently face.

Search is not about getting a list of web pages. The structure of online information is much more complex. If you’re looking for some help on a particular Java library, you wouldn’t expect Google to return you a link to every single page of a 50-page online manual: the whole manual is the information unit you’re looking for. If you’re looking for opinions on a new Wii game, you should expect a list of forums, with for each forum, some insight into how many posts refer to the game, what the overall sentiment is, etc. The posts may be on one same web page, or they may not, and this does not matter.

The concept of relevancy is changing. Google’s strengths was to give you the best results quickly. But if you want to know what is being said about a particular brand, product or person – who’s to say which mention is more relevant than the others? Shouldn’t the relevancy of a mention be a subjective mixture of the credibility, reach, and popularity of the mention’s author? It seems impossible to reduce such a mixture into a single number – and wrong to assign such a number to each and every web page.

The Google index proudly boasts of returning you millions of results, but only ever shows you a thousand (quick tests on the News and Blog searches show you that these numbers are misleading). But the key assumption is that only the top results are interesting. Aren’t they all? Don’t most users want to get a complete picture of the information out there, including information from smaller, less ‘relevant’ sources?

Social content evolves, too quickly for a fat search index like Google’s. Google did create separate products for News and Blog searches, and that is revealing about the limitations of their main search index. Besides, Google Blog search is packed with non-blog content, and full of spam.

The future really seems about extracting ‘keyword mentions’ from web pages, as often as possible, then identifying the mentions’ authors and sentiment, aggregating them in various ways, and so on.

You could argue that Google Search answers a different need: it gives a rather static picture of the knowledge accumulated on the web. This may be why, for half of my searches, the first result is from Wikipedia – in which case, do we really need Google in the first place?

And I subscribe to several blogs including Jason Yip’s which has some nuggety insights.

I was at Reboot this year where Eelco Rustenburg gave a great talk on Agile, Software, Management, India collaboration in which he spoke about the benefits of a more agile approach. During his talk something crystallised in my mind.

Non-Agile development

There are two big problems with setting long release cycles and doing thorough detailed design and specification documents that require the developer to think through every aspect of a development before starting.

1. We think in terms of days and weeks not months and years

Try to think of something that might take a year to do. Build a house maybe? Or write a book? These are broad tasks and it’s difficult to know whether they will take a year or maybe go on for three years. The act of thinking through what needs to be done is extremely difficult and even when that’s done, you may still end up being 100% over your estimates. We don’t like thinking in terms of years, because evolution hasn’t had a need to teach us those skills. Getting a little anthropological for a moment, for most of history, for us humans job #1 has been to eat enough to get through the day. Short term. Job #2 has been to find a partner and reproduce – weeks, maybe months at worst. Until big projects came along, which in our evolution is recently, there hasn’t been the need to think in years so we’re not very good at it.

2. We like gratification

You sometimes have to wait a long time in non-Agile development to get this fix. But you get it much more with an Agile approach. Outline a feature, plan it quickly, prototype it, check it with colleagues and customers, amend it, ship it. 2 weeks or maybe a month, then satisfaction and the sense of job done. That’s a great feeling. My friend Katja puts it nicely – ‘we all need some sugar from time to time’

Agile gives us what we like

So agile development is more in line with our natural event horizons and it gives us the kind of feedback that we crave. What crystallised in my head during Eelco’s talk was that because of these two things, Agile is a primarily a great motivational tool. And motivation is a big factor in productivity.

How hard can that be? It turns out, way harder than it should be. One word aarrrgghh.

Here’s what I did.

The data is the number of pages on the internet as advertised on the Google homepage

* this one is a guess as they stopped publishing the numbers in sept 05 (8.1bn)

Open Office
First up – I tried to generate a chart using Open Office – I’m a Mac user and I don’t have Excel installed as it’s not great on the Mac. My approach was to save as html and go and find the jpg/png that is generated.

Here is the output for open office

One word: BLURRY. And I saved this as the highest quality I could.

Time: That took me about 30 minutes once I had uploaded it to wordpress and previewed etc. Also, the scale on the x axis did not work properly (the gap between 04 and 07 looks way too smail), but I can fix that in the application.

Verdict: 2/10 (once i fixed the x axis issue)

Excel
Next up was Excel – I used one of the other machines in the office which runs XP (SP2) and Excel 2000. Same data. Same approach. Different result

One other word – DARK

Time: about 25 mins including getting the data onto the XP machine

Verdict: 0/10 Unusable

Google Docs
What about the promise that is Google Spreadsheets I hear you cry (I have good ears)

The approach was a little different. Once I created the chart which is called a ‘Gadget’ in Google docs, I ‘published’ it and they gave me a piece of code to put into my page. Not that difficult, but of course the WordPress WYSIWYG editor didn’t like it so I’m forced into editing html directly which is OK for me as I’m down on that stuff, but it’s not accessilbe for non-html folks.

There were several issues with this one. The biggest was that there are very few layout controls. Look at the dates – there’s no way to fix that. The title has disappeared and we have the same x axis scale problem as the Open Office chart but unlike OO, with Google Docs there is no way to fix it.

Time: 30 minutes a lot of which was spent cleaning up WordPress-inserted html

Verdict: 1/10 quite pretty, but not very good and the html issue was a pain
(would have been 2/10, but I just found out that as the graph is stored by Google and linked with the scipt, it doesn’t show up in the RSS feed which sucks)

What about some online app that could generate a flash file? That thought flashed through my head and I let it fly away without trying to stop it and find out more. Bad idea – just a bad idea.

So what do I do?
I get frustrated. I speak to our designer who says she can do it in Illustrator. Illustrator?? This is data!

Not a good use of her time.

Then another colleague suggested taking a screenshot of the Open Office or Excel Version. For some reason that felt all wrong. I had the same feeling I used to get when I taped the top 40 off radio 1 with my radio/cassette player on a Sunday evening 20 years ago.
But, you know, it worked.

The chart is actually a screenshot of the Open Office graph taken with my trusty Mac.

Time: 5 mins plus the original 30

Verdict: 5/10. OK, but not very C21.