Here’s a quick post from under the radar… Apparently, folk from Cam Libraries get together every so often for an informal but issues related brown bag lunch somewhere… It seems like the where and whenabouts of these events is a closely guarded secret.

I think I’m ‘presenting’ at a brown bag lunch session next week, Nov 27th, but I don’t have access to the mailing list the announcement went out on so don’t know any more details than that.

i did, however, manage to grab a bootleg of a trailer for the what may or may not be this event based on what I think I said I could talk about if I managed to get an invite:

If the event is on, I guess I’ll be told immediately before the event and taken to the location blindfolded (presumably using a brown paper bag?)

Just in case, best keep this hush hush, okay? ;-)

As well as a presenting at Online Information this year, I’m also moderating a session on The realtime web: Discovery vs. Search.

To try and frame what I think this topic means to me, I jotted down a set of questions that I’m not sure I have any answers for that I hope will, at least in part, be covered by one or more of the speakers:

Over the last few weeks, the major search engines all announced realtime search capabilities – but what is ‘realtime search’?
 
To what extent can traditional search engine techniques for determining relevancy and ranking results “just cope” with real time signals? Will realtime search focus on ‘turning up’ real time ‘atomic’ status updates, or will it focus on using realtime information as part of a ranking algortihm applied to more traditional web pages? Will ’social discovery’ or social amplification factors just provide yet another ranking factor to traditional search engines, or is it more complicated than that?
 
In financial markets, being able to publish price information in as near as realtime as possible is key to success, but may also require dedicated hardware and high speed/low latency networks. How ‘real time’ does the real time search’n'discovery web really need to be? Will speed be a determining factor in which ‘realtime’ search engine people use?
 
For som time, it’s ben possibility to identify trends in search behaviour with a variety of periodicities (eg Trendspotting). To what extent does/could/should the ‘periodic web’ influence the results that search engines return. If certain signals tend to lead particular behaviours (eg a spike in a real timee signal on a Thursday predicts a certain bhaviour on a Friday, or a cheer of “goal” on twitter predicts and spike in electricity generation as everyone goes off to boil the kettle and make a cup of tea), might that affect not only search engine rankings in ‘real predictive time’, but also other instrumented systems (such as AdWord pricing, or even energy supply)?
 
To what extent is the realtime web just ‘high frequency’ noise (low effort to produce, quick to disappear) compared to more substanital and expensive to produce ‘low frequency’ signals such as the steady accretion of long lasting links to a web page over time?
 
The web is the eleephant in the room, and it nevr forgets. Given we can now capture, and potentially store, ever increasing amounts of real time sourced data, are w going to need ‘forgetting algorithms’? If so, what might those algorithms do, and over what timescales might they operate?
 
With the increasing instrumentation of the web we are seeing a rise in live “operations” data on the web via services such as Pachube (as well as monthy, quarterly or annual data dumps, such as those released increasingly by government here in the UK, in the US, in Australia and so on). To what extent, if any, might this real time machine collected data about our environment play a role in supporting the (public) discovery of real time events in near real time. (So for example, road traffic information, travel information, weather warnings, earthquake warnings etc)
 
To what extent might realtime data in one medium influence discovery in another – for example, if a lot of photos tagged in a similar way are uploaded to flickr with a particular location, how might that signal; be used?
 
To what extent do the discovery of events in real time impact on the enterprise. What sort of role, if any, is there for a real time, or real time supported capability within a corporate/enterprise/intranet search engine?
 
In th UK, an increasing amount of Linked Data is being made available from government sources. What role, if any, does real time sourced data have to play in the discovery of, or rasoning across, Linked Data? What risks might there be to Linked Data systems by the inclusion or availability of dynamic data, or data that is contiinually generated in real time.

And who are the speakers in the session?

• Stephen Arnold, President, Arnold Information Technology, USA
• Antonio Gulli, Principal Development Manager, STC Europe, Microsoft
• Conrad Wolfram, Co-Founder, Wolfram Research and Wolfram|Alpha, UK
• Morgan Zimmerman, VP Business Development, Exalead, France

 
If there are any other questions that you think need asking, please add them as comment below…

I’m not sure how I feel about this – maybe the magic is good magic, maybe it’s voodoo magic, or maybe it’s fake magic, the work of a charlatan, but I wonder, I wonder, might Google’s ‘Personalised Ranking’ utility in Google Reader be useful in filtering, or at least ranking, latest issue table of contents feeds from somewhere like TicTocs?

Only have a 10-minute coffee break and want to see the best items first? All feeds now have a new sort option called “magic” that re-orders items in the feed based on your personal usage, and overall activity in Reader, instead of default chronological order. Click “Sort by magic” under the Folder Settings menu of your feed to switch to personalized ranking. Unlike the old “auto” ranking, this new ranking is personalized for you, and gets better with time as we learn what you like best — the more you “like” and “share” stuff, the better your magic sort will be. Give it a try on a high-volume feed folder or All items and see for yourself!

[Google Reader Personalised Ranking]

Now I believe that there is also a JISCRI project looking at a related sort of thing – Bayesian Feed Filter…: “The Bayesian Feed Filtering project will be trying to identify those articles that are of interest to specific researchers from a set of RSS feeds of Journal Tables of Content by applying the same approach that is used to filter out junk emails.” [Project Kicks Off]

So I’m thinking: it’d be great to see how their approach might filter subscribed to feeds bayesed (!;-) on what users read from those feeds, compared to the Google magic?

For many years now, it’s been possible to subscribe to persistent (“saved”) Google News searches and so build up your own custom dashboard views of news… Indeed, it was over three years ago now that I hacked together a demo news feed roller (Persistent News Search OPML Feed Roller) that let users bundle up a roll of feeds in an OPML file (sort of!) for easy viewing elsewhere.

And if OPML isn’t your thing, then services like Netvibes or Pageflakes let you easily wire up your own news dashboard:

But we all know in our heart of hearts that RSS and Atom feed subscriptions are just not popular widespread as a consumer technology. Folk aren’t knowingly using feeds, and they not unknowingly using them directly either. (But feeds are being used as wiring/plumbing behind the scenes, so RSS is not dead yet, okay?!;-)

(In the Library world, as well as the wider news reading world, this failure to engage with feed subscriptions can be seen (in part) by the lack of significant uptake of RSS alerts.)

So when Google announced last week that you can now Create and Share custom News sections, it struck me that they were getting round the exposed plumbing problem that subscribing to a feed implies, and instead making it easy to create a custom view (the output of which can also be subscribed to) with the appearance of having to do much plumbing at all – How to Create Your Own Google Custom News Section (Tutorial):

You can search the directory of already created news sections – as well as find a link to a page that lets you create your own news sections, here: Google News: Custom sections directory.

So for example, here are a few I have already made:
- UK Higher Education News
- Isle of WIght News
- UK Broadcasting News
- Formula One News

The extent to which you can create a finely tuned view of the news is, admittedly, limited. You can’t, for example, limit the search to specified publications (which you can do in a Google news advanced/search limited search) – filtering is limited to keywords and locale (I’m not sure of the extent to which the order in which you enter the keywords affects things?). But if you already know how to create that sort of filtered search, you probably also know how to set up a new search alert, wire up an feed powered dashboard of your own, and so on. And if the Google Custom News sections editor was any more complicated, I dare say it would put off the users I imagine Google are reaching out to…

As we come to the final month of the JISCPress project, we had some great news over on WriteToReply last week where we were able to announce that Eduserv would be covering our hosting costs for the immediate future (Eduserv funds hosting for WriteToReply, eFoundations: Write To Reply).

So what exactly does the platform we’ve been working on have to offer? Here’s one of the ways I think of it…

A document publishing platform that automatically atomises documents to the paragraph level, allows aggregated commenting at the paragraph and ‘user’ level, and supports the republication and re-presentation of documents in a variety of standard formats at the document level.

The first part of the process is the (manual assisted) ingress stage, in which documents are imported into the WordPress environment such that each substantive document section ideally maps onto a single WordPress “blog post”:

An RSS for the document as a whole, with one item per section, is generated automatically by the WordPress platform. A single item RSS feed is also generated for each page (so the content of each page can be easily transported around the web).

The second part of the process is the atomisation of each post, carried out automatically by the Digress.It theme, in which each paragraph in the document is given its own unique URI, derived from the URI of the web page (“blog post”) the paragraph appears on:

Potentially, an RSS feed can also be produced for each page in which each paragraph is a separate feed item, thus allowing a page/section to be transported around the web via a single feed, but in atomised form.

The paragraph level chunks produced by the atomistation process can be transcluded as independent elements in independent web documents in other documents by a variety of means (as an embeddable object, via XML, txt, JSON, etc):

The default nature of the WordPress platform allows comments to be made at the level of each web page, with an RSS feed of comments for each page being published ‘for free’. JISCPress extends this functionality by allowing comments to be associated with discrete paragraphs. Views over the comments are also available at the user level, (that is, grouped according to the user who made the comments, wheresoever they are made in the document). An additional RSS fed of comments by user is also available, which means that a document on the platform can actually be used as a scaffold for a critical response to the document by a particular user.

A further level of innovation is based on the automated generation of ’semantic tags’ at the page level. Once generated, tag based collections of posts can be syndicated in the normal way via WordPress generated tag based RSS feeds:

JISCPress also benefits from the Trackback mechanism implemented by WordPress. When a page or paragraph URI is linked to from a third party web page, a trackback to the originating page may be captured, which we interpret as the automated capture of links remote annotations or comments about the document.

When considered in these terms, the JISCPress/WriteToReply platform is seen to provide a powerful means of publishing documents in which individual sections may carry their own unique URI, and individual paragraphs within a section also contain their own unique URI (which in many situations may be rooted on the section URI).

The platform can also be regarded as republishing – or re-presenting – each section (i.e. page) and each paragraph as an independent entity. That is, whenever a document is published via the platform, each separate paragraph may also be thought of as being independently published “for free”, in the sense that:

- each paragraph is independently addressable,
- each paragraph is independently commentable, and
- each paragraph is independently republishable/syndicatable.

So, given that, can you think of any ways in which the JISCPress/WriteToReply platform can support your document publishing and comment gathering strategy?

So it feels as if I haven’t been posting that much on this blog over the last few weeks, but I have been blogging elsewhere, 2-3 times a week, in fact, on:

- the Arcadia Project Blog;

- the Arcadia Mashups Blog.

Here’s a quick round up of some of the more notable posts that you can find over there that I would, in the normal course of events, have probably posted here on OUseful.info:

• Do Libraries Cater for Today’s Researchers and Research Students?, a quick response to a new interim project report from the British Library reporting on a longitudinal study about the information related behaviour of research students; [Arcadia Project]
• Open Library Training Materials and Custom Search Engines, in which I describe a simple Google Custom Search Engine that searches over all the UK HEI Library websites. If you’re looking for infoskills/Library training resources, this might be a handy way of finding them… [Arcadia Mashups]
• Getting Started With Yahoo Pipes: Merging RSS Feeds, an quick guide to creating your first Yahoo Pipe [Arcadia Mashups]
• Stack Request Delivery Slots: the Cambridge University Library has many of its borrowable books on closed stacks, which means you have to request the books and then they are fetched for you. Might it be an idea to allow bookable collection slots, so you can request an item for a future date? [Arcadia Project]
• BookServer – Like URIPlay, but for Books…? and LibraryDNS, cf. RadioDNS: Books that Can Phone Home, where I pinch a couple of ideas from online video (content negotiation) and digital radio (linking radioplay back to the web) and start to apply them to books; see also What If? Livescribe Book Support for one possible way of relating written notes to (e)books; [Arcadia Project]
• The Library’s Role in Organising “Course Knowledge”, or how the Library might be the natural place for all manner of content recommendation services to learners; also relates to Looking Out for “Linked Course Data, where I have a poke around various bits of the Cambridge university website looking for unique identifers that can be used to pivot around; [Arcadia Project]
• JISC MOSAIC Competition Entries – Imaginings Around the Use of Library Loans Data, a brief review of the JISC MOSAIC (Library loans data) competition entries;
• ‘Get Current URL’ Bookmarklet Pattern and ‘Get Selection’ Bookmarklet Pattern, in which I describe a couple of bookmarkelt patterns that have generic reuse value; each post includes a bookmarklet generator to help you create your own bookmarklets [Arcadia Mashups]
• Visual Links – Sharing Links With QR Codes, or what might a library catalogue look like if we added a QR code to it; also includes a demo of Cambridge Library short link service that could provide the first step in making it easier to scribble down a book reference [Arcadia Mashups]

They’re all Library related, so if that’s your thang, maybe worth a read…?

Although I’m now half way through my Arcadia Fellowship (sigh….:-(, it wasn’t until last weekend that I spent my first weekend in Cambridge, and finally got around to doing some culture stuff (a couple of galleries, a recital, an excellent lunch in Michaelhouse (thanks for the tip, Huw:-), and so on…

It also made me realise how I haven’t really got into the swing of making the most of my time here, so over the next few weeks I intend to check out the various Cambridge events calendars (of which there are several – more about that in an Arcadia post somewhen…) and start getting some events in…

In fact, I’ve already started, writing this as I am having just got back from a talk tonight by science communicator (and presenter of Material World, Thursdays, 4.30 pm, BBC Radio 4, also on podcast ;-), Quentin Cooper.

This (public) talk, on public perceptions of scientists, was one in a series arranged by CSAR, the Cambridge Society for the Application of Research (events listing), and just one of many dozens of public talks listed on the talks.cam website (again, I’ll write more about that in a forthcoming Arcadia post).

Ever an entertaining speaker, Quentin described the various stereotyped views of “scientists” (lab coat, mad hair, glasses, a crazy smile, and bubbling test tubes and bunsen burners everywhere), as well as suggesting a little experiment for us all to try at home: search for the word scientist in Google image search…

(Turning Safe Search seems to have very little effect (on the front page of results, at least…). Trying the same thing in locale specific versions of Google image search using the local word for scientist is apparently also illunimating…!)

You can also try it with “face search” switched on:

(Just by the by, here are image searches for engineer (face search), technologist (face search).)

Another interesting observation came from a BA web survey that had asked people to name their favourite on-screen scientists. The ambiguity in the question, unsuspected when it was first posted, lead to the majority of answers relating to fictional scientists rather than science/scientist presenters (Why Dr Who beats Einstein these days).

How scientists portray their own work was also on the agenda – and as I’ve long believed, sometimes a little help from the arts can help. One particular set of examples came from the Cape Farewell project, a “cultural response to climate change”, in which various cohorts of (notable) scientists, artists and musicians went off to see the effects glacial melting for themselves. Sometimes it’s the most obvious things that catch you completely by surpise – like the observation that as glaciers retreat, they might uncover islands that have been previously unmapped, an idea picked up by artist Alex Hartley in his piece Nowehere Island.

Anyway, here are a couple of random thoughts I came away from the event with…

When’s someone going to write a drama like This Life or The Office (or pushing it, No Angels, Teachers, Party Animals, A Very Peculiar Practice etc etc) based in a lab/hi-tech factory, where a bunch of “scientists” (as in sci/tech/eng/maths) folk just get on with the everydayness of their working life in a home and work context? Or has there been one and I’ve missed it?

Folk attending the talk were given the option of taking away an attendance certificate for CPD purposes. If I was an informal learner, could I use such an attendance certificate in partial fulfilment of a more formal academic something?

All in all, a good night out; and another one upcoming tomorrow [i.e. on Tues Nov 3rd 2009]: Thinking Like a Dandelion: Cory Doctorow on copyright, Creative Commons and creativity.

So over the last few days, there’s been so much chat around the roll out of official Twitter lists that some people have probably even blogged about them. Ad hoc lists (aka ‘groups’) have been available for some time on a variety of twitter clients, of course, but now there’s ‘central support’ so it seems like everyone is hyping around them (OMG, OMFG, Twitter has lists. Lists! etc.)

I’m still waiting on the API (though a draft spec has been posted) so I can have a go at creating lists automagically from hashtag groups (various posts), but in the meantime, here are a couple of little toys that try to spot what’s going on within the context of a particular list.

First up, a pipe that will look to see what hashtags are being used by folk on a particular list. The pipe feeds off a list of tweets by list members (using the URI pattern http://twitter.com/USERNAME/lists/LISTNAME/statuses.xml ) and then reuses pipework from Searching for Twitter Hashtags and Finding Hashtag Communities, replacing the search elements with a list feed:

So what? So you can enter the URI of a list and see what hashtags folk on that list have been using recently – Twitter list hashtags pipe:

A second, related pipe (this time reusing the Twitter name search pipe (that looks for people who’ve been tweeting particular search terms) and the Twitter names atomiser (i.e. tokeniser) pipe) will look for the names of folk who are part of a recent conversation on the list by virtue of being @d at… Twitter list conversants pipe:

One thing to note about each pipe – make sure you enter the URI in the correct form – http://twitter.com/USERNAME/lists/LISTNAME

If I get a chance, I’ll harden the pipe with a regexp to defend against the missing lists path element; but at the moment, if you donlt use the correct the URI pattern, the pipe will break…

That’s all…

PS the pipe should be hardened now to accept URIs of the form http://twitter.com/USERNAME/LISTNAME

Picking up briefly on Peter Murray Rust’s exhortation to the keynote attendees at ILI2009 that libraries must rediscover Ranganathan’s Five Laws of Library Science to their heart if they are to survive:

I thought I post some free association thoughts on what the five laws say to me. Note that I’m not a librarian, have never studied library science and don’t normally work for the library, though I currently am on an Arcadia Fellowship with the Cambridge University Library. Which is to say, my interpretation may not be the conventional, or accepted one…

So here we go:

Books are for use.
Hmmm… Books are for use… they are they to be used… they exist to be read… they exist to impart knowledge, information, emotion. They exist to communicate. As such, maybe they are social objects? But maybe also, they contain information or knowledge that enables things to be done, ideas to be understood? Maybe they are the next step in helping us do something, achieve something?

In a 2003 blog post outlining ideas for what was to become The Search: How Google and Its Rivals Rewrote the Rules of Business and Transformed Our Culture, John Battelle describes Google’ssearch operation as a database of intentions:

The Database of Intentions is simply this: The aggregate results of every search ever entered, every result list ever tendered, and every path taken as a result. It lives in many places, but three or four places in particular hold a massive amount of this data (ie MSN, Google, and Yahoo). This information represents, in aggregate form, a place holder for the intentions of humankind – a massive database of desires, needs, wants, and likes that can be discovered, supoenaed, archived, tracked, and exploited to all sorts of ends. Such a beast has never before existed in the history of culture, but is almost guaranteed to grow exponentially from this day forward. This artifact can tell us extraordinary things about who we are and what we want as a culture.

That is, every search we make is an expression of some sort of intention. There is a point to every search.

So maybe in the same way, a book might be able to satisfy some intention? Or maybe I’m getting ahead of myself, because second up we have:

Every reader his [or her] book.
So at any point in time, there is a book that I need, that will somehow “help”? This ties back to a book that can satisfy an intention I have, perhaps? My current problem, or situation, is unlikely to be one that has never been met before, never been addressed by someone, somewhere, in some particular book?

Every book its reader.
And conversely, at any point in time, for every book there is someone who would benefit from reading that book? The book is a satisfaction of some intention? There is someone who would benefit from being recommended that book, maybe? (The ideal search engine would be an answer engine, would return only the single answer you need for a particular query, maybe?)

Save the time of the User.
Which means what? Give them the book that they need, in a timely fashion? Make it easy for them to discover the right book, or the right part of the book, that they need, with the minimum of fuss, or noise in the recommendations? Give them full text search, extended indexes in the form of semantic tags and on-demand access, maybe?!;-)

The library is a growing organism.
The library is a living thing. As a living thing, it must adapt to survive. As a living thing, it inhabits an ecosystem, a network, a network characterised by the making and breaking of new and old connections, by the flow of resources across those connections.

Hmmm… so how are these laws actually interpreted by the Library Science community, I wonder? And to what extent do they apply in the context of search engine queries, results and the resources pointed to by those results? Would it be fair to say that it is Google, rather than Library, that has taken these laws to its heart? Would it be fair to say that several of the laws at least hint at making effective recommendations to users, as Lorcan Dempsey suggests in Recommendation and Ranganathan?

Some time ago, I started exploring how treemap representations might be used to provide a quick overview of the make-up of the membership of local council committees (Glanceable Committee Memberships with Treemaps).

Following a lazyweb request to @countculture about expressing complete council committee membership data from Openly Local (Topical Versions of Local Council Websites… Give Us the Data and We’ll Work It For You), and the rapid fulfilment of that request ;-), here’s a proof of concept about how to use that data to power a treemap from the Javascript InfoViz Toolkit (JIT) to provide a glanceable display of the make-up of Isle of WIght Council committees, colour coded by party:

Each committee is represented as follows:

{children: [
 {children: [],
  data: {
   $area: "3", $pcolor: "purple",members:"Jonathan Francis Bacon, Paul Fuller, Heather Humby, "}, id :"ccl12_0", name: "Independent"
 },
 {
  children: [],
  data: {
   $area: "5", $pcolor: "blue",members:"Ivan Bulwer, Susan Jane Scoccia, Albert Taylor, Jerry White, David G Williams, "},
   id :"ccl12_1",
   name: "Conservative"},
{
  children: [],
  data: {
   $area: "1", $pcolor: "darkorange",members:"Colin Richards, "},
   id :"ccl12_2",
   name: "Liberal-Democrat Group"}
], data: {$area: 9}, id: "ccl12", name: "Licensing and General Purposes Committee"}

With another quick bash at the code, I should be able generate these views on the fly from the JSON data feeds provided on the OpenlyLocal site. (At the moment, the route I take to generate the Javscript object that powers the treemap is a really clunky one:-(

What never ceases to amaze me, though, is how a tweak from one representation of a dataset (that is, the JSON data published by OpenlyLocal), to another (the JIT treemap representation) allows the creation of interactive visuliastions as if by magic :-)

If you want to play with your own treemaps in the meantime, this bit of Javascript will produce a simple representation of committee. member and party data that can be visualised within Many Eyes WIkified:

for (var i=0;i< c.committees.length;i++){
  for (var j=0;j< c.committees[i].members.length; j++){
  	document.write("\""+c.committees[i].title.replace("&", "and")+"\""+","+"\""+c.committees[i].members[j]["first_name"]+" "+c.committees[i].members[j]["last_name"]+"\""+","+"\""+c.committees[i].members[j]["party"]+"\""+"<br/>");
  }
}

(where c is the javascript object that is published as the JSON feed from a committee page on OpenlyLocal, such as this one for the Isle of Wight).

Here’s an example:

Whilst this treemap doesn’t allow you to fix the colours in the way that the JIT component does:

TM.Squarified.implement({
   'setColor': function(json) {
     return json.data.$pcolor;
   }
 });

($pcolor is a variable I set for each committee member saying what colour should be displayed for them…), the Many Eyes Wikified does allow you to manipulate the tree representation that powers the treemap, e.g. by reordering the way in which the different elements are displayed:

What I really need now is a way of creating the hierarchical JIT objects on the fly from a table based representation… Hmmm….