Once you have selected a store, you can browse resources by type (rdf:type), search across the contentbox index, edit resources, view pending jobs and send new ones, import data, and configure the field-predicate mapping for your stores.
Please send bug reports and feature requests to keith dot alexander at talis.com
If you do want a talis store, just ask in #talis on irc.freenode.net, or email danny dot ayers at talis.com
My plan is to try and convert it into meaningful RDF and see what we can find out about things along the way. The project is an own-time project mostly, so progress isn’t likely to be very rapid. Let’s see how it goes. I’ll diary here as stuff gets done.
To save me typing loads of stuff out here, today’s source code is tagged and in the n2 subversion as day 1 of OpenLibrary.
Day one, 3rd October 2008, I downloaded the authors data from OpenLibrary and unzipped it. I’m also downloading the editions data from OpenLibrary, but that’s bigger (1.8Gb) so I’m playing with the author data while that comes down the tubes.
The data has been exported by OpenLibrary as JSON, so is pretty easy to work with. I’m going to write some PHP scripts on the command line to mess with it and it looks great for doing that.
Each line of the JSON in the authors file represents a single author, although some authors will have more than one entry. Taking a look at Iain Banks (aka Iain M Banks) we have the following entries:
{"name": "Banks, Iain", "personal_name": "Banks, Iain", "key": "\/a\/OL32312A", "birth_date": "1954", "type": {"key": "\/type\/type"}, "id": 81616}
{"name": "Banks, Iain.", "type": {"key": "\/type\/type"}, "id": 3011389, "key": "\/a\/OL954586A", "personal_name": "Banks, Iain."}
{"type": {"key": "\/type\/type"}, "id": 9897124, "key": "\/a\/OL2623466A", "name": "Iain Banks"}
{"type": {"key": "\/type\/type"}, "id": 9975649, "key": "\/a\/OL2645303A", "name": "Iain Banks "}
{"type": {"key": "\/type\/type"}, "id": 10565263, "key": "\/a\/OL2774908A", "name": "IAIN M. BANKS"}
{"type": {"key": "\/type\/type"}, "id": 10626661, "key": "\/a\/OL2787336A", "name": "Iain M. Banks"}
{"type": {"key": "\/type\/type"}, "id": 12035518, "key": "\/a\/OL3127859A", "name": "Iain M Banks"}
{"type": {"key": "\/type\/type"}, "id": 12078804, "key": "\/a\/OL3137983A", "name": "Iain M Banks "}
{"type": {"key": "\/type\/type"}, "id": 12177832, "key": "\/a\/OL3160648A", "name": "IAIN M.BANKS"}
In total the file contains 4,174,245 entries. First job is to get a more manageable set of data to work with. So, I wrote a short script to extract 1 line in every 10 from a file. The resulting sample author data file contains 417,424 entries. This is more manageable for quick testing of what I’m doing.
So now we can start writing some code to produce some RDF. Given the size of these files, I need to stream the data in and out again in chunks. The easiest format I find for that is turtle which has the added benefit of being human readable. YMMV. Previously I’ve streamed stuff out using n-triples. That has some great benefits too, like being able to generate different parts of the graph, for the same subject, in different parts of the file then being them together using a simple command line sort. It’s also a great format for chunking the resulting data into reasonable size files as breaking on whole lines doesn’t break the graph, whereas with rdf/xml and turtle it does.
So, I may end up dropping back to n-triples, but for now I’m going to use turtle.
I also like working on the command line and love the unix pipes model, so I’ll be writing the cli (command line) tools to read from STDIN and write to STDOUT so I can mess with the data using grep, sed, awk, sort, uniq and so on.
First things first, Let’s find out what’s really in the authors data. Reading the json line by line and converting each line into an associative array is simple in PHP, so let’s do that, keep track of all the keys we find in the arrays and recurse into the nested arrays to look at them - then dump the result out. The arrays contain this set of keys:
alternate_names alternate_names alternate_names\1 alternate_names\2 alternate_names\3 bio birth_date comment date death_date entity_type fuller_name id key location name numeration personal_name photograph title type type\key website
So, they have names, birth dates, death dates, alternate names and a few other bits and pieces. And they have a ‘key’ which turns out to be the resource part of the OpenLibrary url. That’s means we can link back into OpenLibrary nice and easy. Going back to our previous Iain Banks examples, we want to create something like this for each one:
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix bio: <http://vocab.org/bio/0.1/> .
@prefix foaf: <http://xmlns.com/foaf/0.1/> .
<http://example.com/a/OL32312A>
foaf:Name "Banks, Iain";
foaf:primaryTopicOf <http://openlibrary.org/a/OL32312A>;
bio:event <http://example.com/a/OL32312A#birth>;
a foaf:Person .
<http://example.com/a/OL32312A#birth>
bio:date "1954";
a bio:Birth .
This gives us a foaf:Person for the author and tracks his birth date using a bio:Birth event. While tracking the birth as a separate entity may seem odd it gives the opportunity to say things about the birth itself. We’ll model death dates the same way, for the same reason. I’ve written some basic code to generate foaf from the OpenLibrary authors.
Linking back to the OpenLibrary url has been done here using foaf:primaryTopicOf. I didn’t use owl:sameAs because the url at OpenLibrary is that of a web page, whereas the uri here (http://example.com/a/OL32312A) represents a person. Clearly a person is not the same as a web page that contains information about them.
The only thing worrying me is that the uris we’re using are constructed from OpenLibrary’s keys. This makes matching them up with other data sources hard. Matching with other data sources requires a natural key, but there’s not enough data in these author entries to create one. The best I can do is to create a natural key that will enable people to discover the group of authors that share a name.
@prefix mine: <http://example.com/mine/schema#> .
<http://example.com/names/banksiain>
mine:name_of <http://example.com/a/OL32312A>;
a mine:Name .
These uris will enable me to find authors that share the same name easily, either because they do share the same name or because they’re duplicates. The natural key is simply the author’s name with any casing, whitespace or punctuation stripped out. That might need to evolve as I start looking at the names in more detail later.
Next step is to look in more detail at the dates in here, we have some simple cases of trailing whitespace or trailing punctuation, but also some more interesting cases of approximate dates or possible ranges - these occur for historical authors mostly. The complete list of distinct dates within the authors file is in svn. If you know anything about dates, feel free to throw me some free advice on what to do with them…
To use httpclient with Moriarty you just need to ensure that http_class and sasl_interact_class are loaded before using any HTTP actions. Adding lines like the following to your index.php (or somewhere similar) should do the trick:
require_once '/path/to/moriarty/lib/httpclient/http.php';
require_once '/path/to/moriarty/lib/sasl/sasl.php';About Moriarty… Moriarty is a simple PHP library for accessing the Talis Platform. It follows the Platform API very closely and wraps up many common tasks into convenient classes while remaining very lightweight. It also provides some simple RDF classes that are based on the excellent ARC2 class library. Moriarty is primarily being developed by Ian Davis and is in continual alpha, subject to occasional rapid bursts of change. You can read more about Moriarty on the n² wiki or visit its Google Code project
To generate a batch changeset from 2 versions of an RDF graph, you can use an ARC plugin called Talis_ChangeSetBuilderPlugin.
To use it:
$args = array(
'before' => $before, //can be rdf/xml, turtle, or an ARC simpleIndex array
'after' => $after, //can be rdf/xml, turtle, or an ARC simpleIndex array
);
$cs = ARC2::getComponent('Talis_ChangeSetBuilderPlugin', $args);
$cs_response = $store->get_metabox()->apply_versioned_changeset($cs);
The plugin also relies upon the IndexUtils Plugin. The easiest way to get them all set up is to change to your arc directory and do:
svn co http://n2.talis.com/svn/playground/kwijibo/PHP/arc/plugins/trunk/ plugins
The great practical benefit of doing this is you can then reverse previous ChangeSets to return a resource to its previous state. You can read about one way to reverse changesets on the wiki. You can also now create rollback changesets from Moriarty with the new Rollback class.
To use it:
define('MORIARTY_ARC_DIR', 'arc/');
require 'moriarty/store.class.php';
require 'moriarty/rollback.class.php';
//create a store object
$store = new Store('http://api.talis.com/stores/my_store');
//Instantiate the Rollback class with a sparql service object:
$sparql = $store->get_sparql_service();
$rollback = new Rollback($sparql);
//Call the to_changeset method, with a changeset's uri as the argument
$HTTP_Response = $rollback->to_changeset('http://api.talis.com/stores/my_store/items/1200302910905#self');
// the body of the response is the changeset you need to revert back to the
// state of the resource before the changeset that you have given the URI of
if($HTTP_Response->is_success()){
//submit changeset
$rollbackResponse = $store->apply_versioned_changeset($HTTP_Response->body);
if($rollbackResponse->is_success()){
//relax!
}else{
// throw an error
}
}
If you want to take advantage of the streaming, you just need to extend the ARC2_RDFXMLParser class and overwrite the addT method:
<?php
require 'arc/ARC2.php';
require 'arc/parsers/ARC2_RDFXMLParser.php';
class Streamer extends ARC2_RDFXMLParser {
function addT($s, $p, $o, $s_type, $o_type, $o_dt = '', $o_lang = ''){
var_dump($s, $p, $o, $s_type, $o_type, $o_dt, $o_lang);
}
}
$p = new Streamer();
$p->parse('big-data.rdf');
?>
In this simple example, I’m just var_dumping out the triples as they come in, but of course you should do whatever it is you want to do instead to the triple in that method.
Firstly constants.inc.php has been deprecated in favour of moriarty.inc.php which has less of a potential name clash. constants.inc.php is now just a shell that includes moriarty.inc.php so no code should break. However you should update your applications to include moriarty.inc.php because in some future release I shall be removing constants.inc.php entirely.
This renaming is in preparation for a wider breaking change that I would like to make. Because PHP has traditionally had no namespacing capability the community has adopted library naming conventions to avoid name conflicts. For example, classes in Konstruct are prefixed with k_ (like k_Document) and classes in ARC are prefixed with ARC_ (e.g. ARC2_RDFXMLParser). Moriarty doesn’t do this which leads to a higher chance of naming clashes with client code. The right thing to do in a future release is to rename all the classes. So instead of Store we might have MORIARTY_Store or M_Store. I’d like some feedback on what you prefer so please do comment on this post.
It’s worth remembering that moriarty.inc.php defines the MORIARTY_DIR constant, setting it to be the directory in which moriarty.inc.php lives (this isn’t new, constants.inc.php used to do this). The preferred way of including Moriarty classes is like this:
require_once '/path/to/moriarty.inc.php';
require_once MORIARTY_DIR . 'store.class.php';The major piece of new functionality in Moriarty is HTTP caching support. The Platform supports etags and other related caching headers in many places and for a long time I’ve wanted Moriarty to automatically take advantage of these. I added this support over a period of several weeks, refining it and tuning it so that it could work with the minimum of effort on the client developer’s part. Enabling caching in Moriarty is very simple. Just define a constant called MORIARTY_HTTP_CACHE_DIR and set it to be a valid, writable directory. Moriarty will then start using that directory to cache responses from HTTP requests. For example, add something like this at the main entry point of your code:
define('MORIARTY_HTTP_CACHE_DIR', '/var/cache');Moriarty uses cached etag headers to intercept standard GET requests and turn them automatically into conditional ones. Although it still requires a network transaction, the amount of bandwidth used for a cache hit is very small. This kind of caching is smart. Dumb caching just keeps content for a pre-determined time period and only requests a fresh version when the time period has expired. That means it won’t be aware of any changes in the source until minutes or hours later. This may work well for content that doesn’t change often but causes extreme difficulties for interactive applications that involve updating as well as reading content. Many Platform-based applications use a simple pattern of fetching a current resource description, diffing it with the one entered by a user and generating a changeset to apply to the store. Dumb caching interferes with this by not fetching the true state of the resource description, and to fix it requires close coordination between user-supplied updates and cache invalidation. Conditional GETs avoid this by revalidating the cached content with the source on every request. The result is a slight trade off in performance for better consistency.
If you’re confident that you only need dumb caching then you can switch it on by defining the MORIARTY_HTTP_CACHE_READ_ONLY constant somewhere in your application. Moriarty doesn’t care about the value of this constant, just whether it is defined or not. When this constant is defined Moriarty will use the max-age headers in HTTP responses to determine how long retrieved content should be considered to be fresh for. It intercepts HTTP requests to the Platform and if it finds a fresh cache entry then it will immediately return that without making any network request. If the cache entry is stale then the request proceeds as normal and the entry gets updated with the newly retrieved content. Use this constant when your application is predominantly read-only and you don’t care if content is stale for a few hours.
Moriarty supports one other caching related constant: MORIARTY_HTTP_CACHE_USE_STALE_ON_FAILURE. Define this constant if you want Moriarty to return a cache entry when it can’t communicate with the Platform. This enables your application to continue runng even if there are network problems, a tradeoff of apparent liveness against freshness of content. (I use this constant in my application when I’m developing offline on the train. While I’m on the network I hit a few pages to freshen up the cache and then when I disconnect I can still browse and test the application using cached content.)
One caveat you need to be aware of: the cache files are not encrypted. Avoid using Moriarty’s caching support if you are dealing with private or secure information that you don’t want to be stored unencrypted on a web server file system. I might provide this capability if there is demand.
Finally, this version of Moriarty includes support for the new describe service. This was included as part of release 13 of the Platform and is now the preferred way of obtaining resource descriptions from the metabox or a private graph. See the section labeled “GET” in the Metabox documentation. You can use it in Moriarty like this:
$store = new Store('http://api.talis.com/stores/mystore');
$mb = $store->get_metabox();
$response = $mb->describe('http://example.com/foo');The best thing is that the new describe service supports etags for resource descriptions, which means that Moriarty’s new caching functions can really speed up applications that use describe heavily (and if you’re building open world applications, then you should be). The Platform’s SPARQL services don’t currently support etags, so caching is less efficient. To support efficient HTTP caching we’d need to determine whether the resultset has changed since the last time the client issued the query. The only way to do that in SPARQL is by executing the query which could be very cheap or it could be horrendously expensive. The describe service packages up a very common use of SPARQL into a constrained service that is very easy to relate to changes in the underlying graph. That means we can really optimise this service, provide decent caching support and generally boost performance a lot more easily than we can for arbitrary SPARQL queries. Expect us to expand on the describe service in future releases and also to bite off a few other constrained derivatives of SPARQL.
About Moriarty… Moriarty is a simple PHP library for accessing the Talis Platform. It follows the Platform API very closely and wraps up many common tasks into convenient classes while remaining very lightweight. It also provides some simple RDF classes that are based on the excellent ARC2 class library. Moriarty is primarily being developed by Ian Davis and is in continual alpha, subject to occasional rapid bursts of change. You can read more about Moriarty on the n² wiki and get its source from the n² subversion repository
The scripting workshop itself was excellent - I enjoyed all of the others’ papers a lot, and look forward to playing with the code, ideas, and applications they presented.
The paper just before me was about using RDFa and javascript to allow in-page editing of resources. One particularly nice thing about it was that they implemented our RDF/JSON specification in their API (cheers guys! 
).
The scripting challenge was won by the highly deserving but sadly absent Benjamin Nowack with SPARQLBot - his IRC bot that can read and answer questions from RDF data sources. The second prize was won by Alexandre Passant and co for their Semantic Microblogging system, which looks very interesting indeed.
There were quite a few interesting papers on SPARQL - extending it in various ways, or extending SPARQL into other technologies. There were two papers on using SPARQL to bridge the XML and RDF divide: one on embedding SPARQL in XSLT extension functions (’enabling the developer to combine XSLT and RDF in a way that doesn’t suck’) ; another on combining XQuery and SPARQL.
One paper I thought was especially interesting was about extending SPARQL to work on streams of data.
The best paper award was on SPARQL, and won by Christoph Kiefer, Abraham Bernstein, and André Locher for “Adding Data Mining Support to SPARQL via Statistical Relational Learning Methods”.
Richard Cyganiak presented Neologism - an open source drupal-based web application for creating and publishing vocabularies. What is great about this application is not its features, but its philosophy. While desktop applications like Protégé may provide lots of features for designing, creating and reasoning with ontologies, they don’t help with the publishing of them - which can be a rather tricky issue. The idea behind Neologism is to make it easy for vocabulary authors to do the right thing, and author and publish their vocabularies according to best practice - an aim I really applaud.
I also attended a tutorial on developing ontologies with the use of patterns - starting by reusing some basic modeling patterns, which could be used as a mold and later discarded when the design was complete. The tutorial incorporated this into an XP system of development, involving test-driven ontology design - which I thought was an interesting idea.
One thing from the conference I particularly enjoyed was Michael Hausenblas, Richard Cyganiak, Jun Zhao, and I developing Michael’s idea of metaLOD into a vocabulary for describing datasets like those in Richard’s famous LOD diagram (see the slide in the photo above).
The idea was to come up with a light-weight vocabulary that would enable RDF descriptions of interlinked datasets; these descriptions (and so the various access points to the datasets) can be made discoverable via the Semantic Sitemaps extension, and aggregated via services like Sindice. I have a practical interest in this as well, since with the Platform (and our work on the Open Data License), we want to enable people to publish lots of datasets on the web. We already have lots of interesting datasets in the platform (which I have been doing some work on describing in the silkworm-dev store), and we are really keen to make our publicly available datasets discoverable by machines and humans, and available for reuse.
We spent about an hour discussing the weighty issues of scope, vocabulary reuse, and ontology modeling patterns - then at least a further hour trying to come up with a suitable acronym. Finally Laurian advised us not to think of an acronym, but of a cool-sounding word that encapsulated what the vocabulary would give to the world. So we came up with voiD: a Vocabulary Of Interlinked Datasets.
(see also Orri’s “VOID, Or will the LOD Cloud Bring Rain”)
I saw two papers on games and the semantic web at ESWC. The first was Knud Möller’s highly entertaining talk on World of WebCraft - Mashing up World of Warcraft and the Web at the Semantic Scripting Workshop; where he showed how he gleaned semantics from the game by scripting addons, mashing it up with data from dbpedia, and screenshots from flickr.
The second was Katharina Siorpaes‘ presentation of her work on OntoGame, an application of Luis von Ahn’s Games With a Purpose concept to using online multiplayer games for getting people to perform what might otherwise be rather dull tasks in ontology creation and alignment, and data annotation. The idea centers around using blind collaborative game-play to achieve consensus and accuracy on what is common knowledge to humans, yet opaque to machines. I wondered if such game-play would be compelling to mainstream users, but according to the paper’s authors, the social aspect of these games can provide plenty of interest and incentive to keep playing. It’s a thought-provoking concept anyway, and it will be interesting to see what develops in this area, and in which niches these techniques will work best.
The demos were really good. From asking the other attendees, I think the favourites were QuiKey, a Quiksilver-like interface for entering and searching through triples (which won best poster); xOperator (a really intriguing combination of jabber, SPARQL, and Agents to bring you trusted answers to questions via Instant Messaging); OntoGame (described above); and Konduit, a Semantic Pipes-like application for visual programming for the Semantic Desktop (which made me reconsider my position in an irc discussion with iand about whether to describe application flow in code or data; it also won best demo).
The lightning talks, were, as ever a popular and light-hearted, yet thought-provoking event. The format was a tight 2 minutes, 1 slide, which was strictly adhered to: Andraž Tori of Zemanta gave a very good presentation that was roundly and deservedly booed when he tried to slip in 3 slides. At first I thought 1 slide would be a bit limited, but it was actually pretty good - giving each speaker a chance to present only one single idea at a time. All the talks were entertaining, but some that stood out for me were:
;)
The only great pity was that the lightning talks were run in parallel with other tracks, so I missed out on the start of the Applications track. Next year, hopefully they will run the lightning talks separately from the rest, and record them for posterity (the other talks had video-cameras in attendance).
ESWC is a pretty academic conference, but it was really interesting to meet people from other companies making great use of semantic technologies to their competitive advantage, like ProKarriere, an Austrian online recruitment service that uses tools like Crowbar and Solvent to scrape semantics from partner web-sites, together with Natural Language Processing, and ontologies they have developed, to intelligently match graduates up with appropriate vacancies. Or like Net7, who have built Talia, a semantically backed digital library for Philosophy Scholars (described in their Scripting Workshop paper). Or like Garlik, who had a whole keynote about them.
A theme that appealed to me was using semantic technologies outside of Computer Science departments to aid scholars in other fields - there was quite strong presence from Finland with their work in the cultural heritage sector, visualising time and space. I also really enjoyed seeing Jun Zhao’s presentation about using off-the-shelf semantic software like Exhibit to help zoologists navigate a repository of research images, the Net7 guy’s demonstration of their Digital Philosophy library Talia, and hearing about Norman Gray’s work using RDF with astronomers.
While the quality of the presentations was really high, the best bits, as usual, were the socializing and informal discussions in between, meeting names I’d long been familiar with from the various semweb mailing lists, blogs and irc channels (#swig, #swhack, #sioc etc), and new people besides.
It was also nice to meet a few other people living in Scotland doing semweb stuff - there doesn’t seem to be that many of us. So I set up http://groups.google.com/group/swig-scotland in the hope that we can all arrange to meet up some time and talk triples (please join if it’s of any interest).
Looking over my copy of the proceedings, I realise that there’s so much stuff I didn’t see that I would have liked to (the tragedy of parallel sessions), and so much stuff I did see that I haven’t done justice to here - all the Nepomuk semantic desktop stuff for instance, or DERI’s research into sensors connected to the web, or the Vapour tool for testing HTTP conneg, or … but I have to stop now :). Suffice to say, it was great, and I’ve got a lot to think about and try out over the coming weeks.