Google API Project and Google Chrome extension

First, you need to register your application as an API Project in Google's API Console, and enable the "Drive API" and "Drive SDK" services.

In the "API Access" section, generate an OAuth 2.0 client ID (choose "Web application", and enter the URL of the web application). This should provide a client ID, a client secret, and a URI to which OAuth authentication requests will redirect after confirmation.

In the "Drive SDK" section, fill in all the required information. Use the client ID generated above for the "OAuth Client ID" field. I'm not sure exactly which scopes are needed here, but I've entered "https://www.googleapis.com/auth/drive.file", "https://www.googleapis.com/auth/userinfo.email" and "https://www.googleapis.com/auth/userinfo.profile".

Next, the App ID from the top of the "Drive SDK" section needs to be added to a Chrome extension. Create a Google Chrome extension for your web application, including the settings "container": "GOOGLE_DRIVE" and "api_console_project_id" (the App ID) in its manifest.json, and publish it on the Chrome Web Store.

When a user installs this Chrome extension, the web application asks for permission to access the user's Google Drive space.

PHP Client Library for Google APIs

Google provide an API Client Library in PHP, and if you checkout the current version of the code there is a newly-added apiDriveService class. At the time of writing it still has a few bugs, so this patch needs to be applied.

I've used the API Client Library in a simple PHP script to upload a PDF file to Google Drive. The steps involved are described below.

To use the client library, put the src folder in your include path, and include the appropriate files in a PHP script:

require 'google-api/apiClient.php';
require 'google-api/contrib/apiOauth2Service.php';
require 'google-api/contrib/apiDriveService.php';

Initialise the client, using the OAuth client ID, client secret and client redirect URI generated in the API console:

$client = new apiClient();
$client->setUseObjects(true);
$client->setAuthClass('apiOAuth2');
$client->setScopes(array('https://www.googleapis.com/auth/drive.file'));
$client->setClientId($config['client_id']);
$client->setClientSecret($config['client_secret']);
$client->setRedirectUri($config['client_uri']);

To authenticate the client for this user, first generate an authentication URL:

print $client->createAuthUrl(array('https://www.googleapis.com/auth/drive.file'))

Visit that URL in a web browser, and confirm that the web application is allowed to access your Google Drive files. Copy the code from the end of the URL to which you are redirected after confirmation, and use it as $_GET['code'] to retrieve an access token:

$_GET['code'] = ''; // insert the verification code here
file_put_contents('token.json', $client->authenticate());

Use the stored access token to authenticate the client each time it runs:

$client->setAccessToken(file_get_contents('token.json'));

Create a new Google Drive file, attach the raw data (from "test.pdf" in this case) and upload it to the Google Drive service:

$file = new DriveFile;
$file->setMimeType('application/pdf');
$file->setTitle('test.pdf'); // it's important that the title includes the correct file extension

$service = new apiDriveService($client);
$insertedFile = $service->files->insert($file, array('data' => file_get_contents('test.pdf'), 'mimeType' => 'application/pdf'));

The file should now be available in Google Drive.

Here's what I've done with it so far:

Start a Ubuntu server instance on Amazon EC2 (the "medium" variety works best, or the "high-CPU medium" (dual-core) if you have a script that can run in parallel).

Create a 10GB volume, attach it to the instance and mount it at /marc:

sudo mkfs.xfs /dev/xvdf
sudo mkdir /marc
sudo mount /dev/xvdf /marc
sudo chmod 777 /marc

I also mounted an empty 50GB volume at /mods, for storing individual files transformed to a different format (see below).

The dataset is available as a gzipped tarball containing 10 MARC21-formatted files. Download to a temporary directory using aria2:

aria2c --max-connection-per-server=5 http://openmetadata.lib.harvard.edu/bibdata/data

wget --continue http://openmetadata.lib.harvard.edu/bibdata/data

Once the file has downloaded, extract the MARC21 files: tar -xvzf harvard.tar.gz -C /marc. I've shared an EBS snapshot of this stage (snap-a099a1dd in us-east-1), so you can start from this point by creating a volume from it and attaching it to a running EC2 instance.

Run a PHP script that opens each MARC21 file, converts each record to MARCXML using File_MARC, then transforms each record to MODS, using a stylesheet provided by the Library of Congress. I've also shared an EBS snapshot of this stage (snap-90333fed in us-east-1), as it took a while to run. It happened to leave the MODS namespace off the output XML, which may or may not make it easier to work with…

I've begun one more step, which is an XSL transformation from MODS to CloudSearch input XML - the idea being to import all the data into a CloudSearch instance and make it browsable/searchable there. There are still some fields to add, though. Other transformations that might be useful include Turtle, for import into a triplestore like Kasabi; HTML, for browsable/crawlable individual records; JSON, for loading into ElasticSearch, MongoDB or a JS interface.

]]> tag:hublog.hubmed.org,2010://2.1952 2012-03-12T12:04:57+00:00 2012-03-12T10:44:24+00:00 A tiny bookmarklet for getting from a BBC Radio or iPlayer web page to the XSPF tracklisting, for use in Tomahawk:

Programme XSPF

Note: I've placed all the code and files associated with this post in a repository on GitHub. If you'd like to fork the project and add a script which processes documents through your favourite entity extractor (storing the results in a new directory), I'd be happy to receive pull requests.

Fetching the documents

First of all, find a set of open access documents in a standard XML format. Articles deposited in PubMed Central (PMC) are ideal, as they are converted from publisher-specific DTDs to one of the standard NLM Journal Article DTDs during deposition. PMC also has an OAI interface, which makes it straightforward to find and retrieve articles.

To find the name of a set of articles, use the OAI "ListSets" command to fetch all the sets into a local CSV file. Have a look through that file and find the set you're interested in - in this case I'm using "elsevierwt": Elsevier's "Sponsored Documents", for which a fee has been paid on publication to make the articles open access; the license allows text mining for non-commercial purposes*.

Use that set name with the OAI "ListIdentifiers" command to fetch the identifiers for all documents in that set into a local CSV file. This script checks that each article is also in the "pmc-open" set, which denotes the Open Access subset of PubMed Central.

For each identifier, use the OAI "GetRecord" command to fetch the document XML into a local folder. The document identifier can be base64-encoded into the filename, so it's easy to identify later.

Converting the documents

Convert all the XML files to the most up-to-date NLM Journal Article DTD, using the XSL transformation provided by the NLM for this purpose. In this case, I'm converting from v2 to v3 of the NLM Journal Article Archiving and Interchange format; once JATS becomes the official standard hopefully the same tools will be provided for conversion.

Convert the article metadata from the XML into RDF triples in Turtle, and store them in a Kasabi data set:

find . -name '*.ttl' -exec curl -vvv -H "Content-Type: text/turtle" --data-binary @{} http://api.kasabi.com/dataset/{$STORE}/store?apikey={$APIKEY} \;

Finally, convert the body of the article to simple HTML, using another XSL transformation. All the inline elements will become "span" elements, all the block-level elements will become "div" elements**.

Text mining

Now the articles are ready for text mining. Choose an entity extraction tool or web service and run each article through it. I'm using the EBI's Whatizit here, which has a SOAP web service that understands plain text and returns XML. If you're lucky, you'll have a simple HTTP POST web service that understands HTML and returns JSON.

Store the results locally, and extract the data you need into RDF triples as Turtle. So far, I've extracted disease and protein names from these articles using Whatizit; the easiest way to find the names for the Whatizit processing pipelines is to View Source and look at the options in Whatizit's HTML entry form.

Post the Turtle files to the same Kasabi data set as the article metadata, where they can be browsed and queried using SPARQL:

find . -name '*.ttl' -exec curl -vvv -H "Content-Type: text/turtle" --data-binary @{} http://api.kasabi.com/dataset/{$STORE}/store?apikey={$APIKEY} \;

* This particular license is quite vague, full of restrictions, and doesn't mention what you can do with derivative works - such as the results of text mining. You might want to choose a set of articles from PLoS or BioMed Central instead, which are clearly licenced with Creative Commons CC-BY licences.

** Each element retains its attributes, and a "class" attribute is added for styling if you ever want to display this HTML.

Of the ~1,000,000 articles in PubMed Central, ~180,000 are marked as "Author Manuscript" - deposited in response to funder mandates, or where the author has retained permission to deposit a pre-print in repositories. However, only 163 of these are marked as "Open Access".

The question - raised by Casey Bergman - is why there is so little overlap: why are author manuscripts not being marked as Open Access in PubMed Central?

As a start, every item needs an identifier. The ISSN International Centre assigns ISSNs to publications that request them, but the data file listing all the ISSN:serial title mappings is copyrighted, expensive and not redistributable (it also uses sentence case for journal titles, which means some information gets lost).

As each publication can have multiple ISSNs - one for each medium in which it is distributed (for example, the online version of a publication can have a different ISSN to the print version) - a pan-ISSN identifier is required to link all the ISSNs together, so the ISSN-L was introduced.

The table that maps ISSNs to ISSNLs can be downloaded from issn.org after filling in a form requesting access. In the latest table, there are 1,614,355 unique ISSNs, mapped to 1,552,542 unique ISSNLs.

This ISSNL:ISSN mapping table, like all the information published by the ISSN Internation Centre, is protected by sui generis database rights, which last, according to Wikipedia, for 15 years from the last substantial update.

The databases appearing on or accessible from the website "the ISSN International Centre" are the exclusive property of CIEPS and are protected under the provisions of the law of 1st July 1998 implementing in the Intellectual Property Code the European Directive of 11 March 1996 on the legal protection of databases. Any performance, whether total or partial, of this site by any company whatsoever, without the express authorization of the CIEPS is strictly forbidden and shall constitute an infringement sanctioned such as Intellectual Property Code.

(note, incidentally, that I'm already in conflict with section 3 of the legal notice: "Users and visitors cannot place a hyperlink to this website without the CIEPS' express and prior authorization."…)

The full database itself may be copyrighted, but (I believe) the individual facts within it shouldn't be. In which case, the best non-copyrighted source for the journal title, ISSN and ISSN-L of each serial is probably the publishers themselves, and as many publishers make their metadata through an OAI interface it may be possible to extract a fair amount of serials information from those sources. ScientificCommons, for example, harvests articles from OAI repositories, so may be able to aggregate useful title and ISSN information.

Existing, non-free sources

A commercial source of the information I'm looking to create is the JournalSeek database (from Genamics, licensed through OCLC) which includes around 100,000 journals.

SHERPA/RoMEO aggregates journal lists from Zetoc, DOAJ (7500 journals) and Entrez (40,000 journals), but the data is only available for non-commercial use.

Other sources of Journal/ISSN information

• 300,000 serials in Ulrich's (JSON interface somewhere?)
• SerialsSolutions Summon (title list as PDF)
• 13,000 journals in EBSCO "Academic Search Complete"
• 91,000 journals in WorldCat Local
• 17,000 journals in Thomson Reuters Master Journal List
• 27,000 journals in CrossRef
• 12,500 journals archived by Portico
• 9,000 journals participating in LOCKSS
• 3,330 journals in ScienceDirect
• Journal lists by subject in Bing Academic Search
• Search journals in the CAS Source Index (copyrighted by the ACS)

It would be nice if Freebase could serve as a central repository for this information, but there are only 4,321 journals in Freebase so far.

Abbreviations

Once we have the list of ISSNs and journal titles, we also need the corresponding journal title abbreviations, for use when generating bibliographies. There are lists of journal title abbreviations available for import into EndNote. Sadly, there are several different abbreviation styles (ISO, MEDLINE, BIOSIS, CASSI, etc).

• The ISSN International Centre maintains the list of Title Word Abbreviations which corresponds to the ISO 4 standard (available from the ISO store for ~£50), which describes the rules for abbreviating title words and titles of publications. The list of Title Word Abbreviations is available online as HTML.
• The NLM uses the list of Title Word Abbreviations to abbreviate periodical titles.
• The NCBI provides a less-comprehensive list of abbreviations for commonly-used English words in journal titles.
• The jabbrv LaTeX package abbreviates journal titles using the list of Title Word Abbreviations.
• The JAbbr service built a list of abbreviated serial titles from the Cornell library catalog of MARC records, and provides abbreviation → full title mapping as a JSON and HTML web service (source code provided). Article in Code4Lib Journal.
• The NLM Catalog is searchable using a journal title abbreviation, but uses sentence case for full titles.
• The NLM Catalog is also searchable by various ISSNs.
• Web of Science has a list of journal title abbreviations.
• A large list of Journal Abbreviation Sources.

More links

http://pinboard.in/u:hubpin/t:journals/

<!-- edit this; the PDF file must be on the same domain as this page -->
<iframe id="input" src="your-file.pdf"></iframe>

<!-- embed the pdftotext service as an iframe -->
<iframe id="processor" src="http://hubgit.github.com/2011/11/pdftotext/"></iframe>

<!-- a container for the output -->
<div id="output"></div>

<script>
var input = document.getElementById("input");
var processor = document.getElementById("processor");
var output = document.getElementById("output");

// listen for messages from the processor
window.addEventListener("message", function(event){
  if (event.source != processor.contentWindow) return;

  switch (event.data){
    // "ready" = the processor is ready, so fetch the PDF file
    case "ready":
      var xhr = new XMLHttpRequest;
      xhr.open('GET', input.getAttribute("src"), true);
      xhr.responseType = "arraybuffer";
      xhr.onload = function(event) {
        processor.contentWindow.postMessage(this.response, "*");
      };
      xhr.send();
    break;

    // anything else = the processor has returned the text of the PDF
    default:
      output.textContent = event.data.replace(/\s+/g, " ");
    break;
  }
}, true);
</script>

See an example running as a live demonstration.

It'll only work in recent browsers, as it requires sending binary data between windows as an ArrayBuffer using window.postMessage, and Web Workers in pdf.js.

Basically, this fetches a PDF as an ArrayBuffer using XMLHTTPRequest, then posts it to the embedded window, which uses pdf.js to render the PDF to Canvas (invisibly; you can see the rendered images if you poke around a bit with a web inspector tool). As it does so, an HTML layer is constructed, containing a block to match each row of the PDF - this would normally be overlaid on top of the rendered images to allow text to be selected, a technique used by many services that allow PDF text selection and highlighting, including Crocodoc and Google Docs' PDF viewer. By taking the text content of those blocks, the service can return the contents of the PDF as a single block of text.

I expect that pdf.js will acquire a native function for retrieving the text content directly, to make documents searchable. It would be nice, next, to try to recreate paragraphs by looking at the spacing between the blocks, and to use the formatting and other heuristics to extract metadata like title, authors, etc.

We now have a standard way of providing metadata about any object, based on two principles:

1. Every object is represented by at least one HTML page on the web.
2. Properties of that object are represented as <meta> elements in the <head> section of that HTML page.

From that, we can make statements about any object using URIs, and fetch metadata about that object using HTTP. The Semantic Web!

Statements

This is an RDF statement:

That's two things connected by a link, all represented by URIs.

Creating a Graph

Several of this kind of statement can be combined to make a graph:

Or, to write those statements in shorthand, without repeating the first part of each one:

And using prefixes to avoid having to write out the full URI each time:

Fetching information

The URI <http://music.com/band/nirvana> represents the band Nirvana. We could equally have used <http://en.wikipedia.org/wiki/Nirvana_(band)> or <http://open.spotify.com/artist/6olE6TJLqED3rqDCT0FyPh>*. As these are HTTP URLs, a representation of this Thing can be fetched using HTTP - in this case, your web browser probably receives an HTML representation of the band.

How does the server decide in which format to return that information? There's a negotiation between whoever requests the information and the server that provides the information. The request contains a list of formats that it would be able to handle, and the server returns the first of those that it's able to provide.  In fact, the information about a Thing might be available as JSON, or XML, or any other format, but Open Graph requires that every Thing identified by a URL must have an HTML web page that represents it.

In this way, we can make statements about any Thing, and fetch information about that Thing by dereferencing its URL to see what information it provides.

How should the information about the Thing be presented in that HTML page**? As the page represents the Thing***, this information can be added to the <head> section of the page; it ends up looking like this:

Which is exactly the same information as in the shorthand RDF statements above. It's RDF in HTML!

If someone says they like the album "Nevermind", a statement is created:
<http://facebook.com/eaton.alf>    <http://example.com/emotions/likes>    <http://open.spotify.com/album/6okv1avxEgYSdc2JYy6ZEi>

When we fetch the HTML document from the URL referenced (<http://open.spotify.com/album/6okv1avxEgYSdc2JYy6ZEi>), it contains (amongst other things) this information:

And when we fetch the HTML document from the "musician" URL <http://open.spotify.com/artist/6olE6TJLqED3rqDCT0FyPh>, it contains (amongst other things) this information:

When all that information is combined, we know that this person, who clicked the "Like" button while listening to an album in Spotify, liked the album "Nevermind" by the musician "Nirvana" - which is what gets displayed in their Facebook timeline.

Referring to URLs

This is all relevant to my recent post about citing with URIs. In that demonstration, the script dereferenced the URI to get information about the thing, but specifically asked for JSON. In the end, though, the JSON is basically just a list of properties about the thing being referenced, and there's no reason why that information can't be represented in <meta> elements in the <head> of an HTML page, which is exactly what most publishers do in order to get their documents indexed by Google Scholar. They use several prefixes ("dc.", "prism.", "citation_"); they often use meta[name][content] instead of meta[property][content], but it's all basically the same thing. I've now updated the script to parse <meta> elements from HTML, alongside JSON responses.

In summary: if someone wants to refer to a Thing, they should be able to use a HTTP URL. If someone wants to get information about that Thing, they should be able to dereference that URL, get an HTML document, look in the <meta> elements in the <head> section, and retrieve all the information about that thing (including further URLs to find out more information about any of those properties).

* Asserting equivalence between URIs allows links from one URI to also apply to the other. For example:
<http://example.com/music/nirvana> <http://www.w3.org/2002/07/owl#sameAs> <http://open.spotify.com/artist/6olE6TJLqED3rqDCT0FyPh>

** We don't have to worry about representing multiple items on a single page - each one will have a link to its own, individual page.

*** We don't have to worry about whether the URI represents the Thing or a document about the Thing: it's always the Thing. Most of the time, no-one cares who wrote the document about the Thing, or when that document was last updated. An exception might be Wikipedia, so I have a suggestion: the Thing is still represented by the web page; information about authors and update times can be attached to an appropriate property of the Thing, e.g. <http://en.wikipedia.org/wiki/Nirvana_(band)#description>.

Citing With URIs

The most straightforward way of being able to cite something in a document is to insert an identifier. On the web we hyperlink using URLs, which provide a unique identifier for the item being referenced - with the added bonus of being able to follow that URL to retrieve the item. When writing a scholarly article, however, there's still an expectation that the metadata for a citation will be provided, so that the reference will still make sense even if the URL stops working.

To be able to successfully cite using identifiers, therefore, means being able to retrieve the metadata for each identifier, and the simplest way to do that is to convert that identifier to a URL - if it isn't already - and retrieve it using an HTTP request.

Once we have the metadata for each citation, all that's needed is to generate a bibliography (a list of endnotes) at the end of the document, and insert links to those references inline. As a complication, there are many different publishing systems, and they each have their own special preferred formatting for those inline citations and bibliographies, so the tool should ideally be able to cater for any of those formats.

There is a need for citation software that works with Google Docs, as it's basically the standard online writing tool (and is continually getting more awesome). I've managed to get the first steps of a citation processor working in Google Docs; it's not complete yet...

Inserting and Processing Citations

Google Apps Script provides a way to add menu items to Google Docs and call a function when a menu item is selected. It's server-side Javascript, with an online editor that functions well. You can currently only attach scripts to Google Spreadsheets, but that's ok in this case: we need somewhere to store a local copy of our references.

Here's how to use Google Apps Script to format citations in a Google Document:

1. Create a new Document in Google Docs and give it a unique title.
2. Write your article, adding citations inline in the form {{cite:doi:10.1038/nchem.1108}}.
3. Create a new Spreadsheet in Google Docs and give it a title which is the same as the document, but with " - References" at the end.
4. Add my Exciting script to the spreadsheet (Tools > Script Editor). Once it's installed, an "Exciting" menu should appear.
5. From the "Exciting" menu, select "Generate Bibliography".

The script will now create a copy of the document (which must be in the same folder as the spreadsheet, and have the same name minus the " - References" suffix). The original document will remain untouched. It will parse the document for {{cite}} strings, fetch the metadata for each one, and store the data in the current spreadsheet (if the script is run a second time, it will use this local data instead of fetching it again). It will then replace the inline citations with numbered references, add a formatted bibliography at the end of the document, email you a PDF of the final, formatted document, and move the formatted copy of the document to the trash.

[NB: this is a first attempt, written last weekend. The citation formatting is very, very basic.]

There are several ways to cite using this system, and this is where it gets most interesting:

• You can use {{cite:doi:10.1038/nchem.1108}} to cite an item by DOI; in this case the data will be fetched from CrossRef.
• You can use {{cite:mendeley:123456}} to cite an item using its ID in your Mendeley library; in this case the data will be fetched from mendeley.com (this might not be working properly yet - I haven't tested it much. It uses OAuth authentication, so you need to register an application and get a key from the Mendeley Developers Portal. You also need to run the "authorizeMendeley" function from within the Script Editor, to authorize this application).
• You can use any URL, as long as it returns JSON when specified in HTTP Accept headers. For example, {{cite:http://dx.doi.org/10.1038/nchem.1108}} works just as well as the DOI example above.

Theoretically, you can cite any URL, and the script will retrieve the metadata from that URL and make use of it. In practice, not nearly as many URLs as I'd like perform content negotiation and return JSON instead of HTML from the same URL, and even when they do there's no standard format for the reference metadata (which is where RDF comes in, but there's no RDF parser in Google Apps Script; RDF triples as JSON would be an good intermediate). The current script has custom functions to normalise the data returned from CrossRef and Mendeley into a single, standard format for local use; adding other sources would probably require a custom parser for their metadata as well.

I'm not able to enter the Mendeley/PLoS API Binary Battle, but |'d be delighted if anyone who's interested was to take this code and make use of it. I see the next steps like this, possibly: 1) get citeproc-node running on a node.js server somewhere (Heroku or Joyent, maybe), and use that for formatting the references; 2) use the UI Services/GUI Builder in Google Apps Script to build an editing interface, for tidying up references once they've been retrieved; 3) add the ability to specify custom formatting for the inline citations, and to choose the citation format for the bibliography.

This means that anyone can now make client-side PubMed search interfaces, like this one.

Only the eSearch and eSummary methods have the Access-Control-Allow-Origin header so far, so it's not possible to get abstracts or full citation data this way (using eFetch) yet.