As a start, every item needs an identifier. The ISSN International Centre assigns ISSNs to publications that request them, but the data file listing all the ISSN:serial title mappings is copyrighted, expensive and not redistributable (it also uses sentence case for journal titles, which means some information gets lost).

As each publication can have multiple ISSNs - one for each medium in which it is distributed (for example, the online version of a publication can have a different ISSN to the print version) - a pan-ISSN identifier is required to link all the ISSNs together, so the ISSN-L was introduced.

The table that maps ISSNs to ISSNLs can be downloaded from issn.org after filling in a form requesting access. In the latest table, there are 1,614,355 unique ISSNs, mapped to 1,552,542 unique ISSNLs.

This ISSNL:ISSN mapping table, like all the information published by the ISSN Internation Centre, is protected by sui generis database rights, which last, according to Wikipedia, for 15 years from the last substantial update.

The databases appearing on or accessible from the website "the ISSN International Centre" are the exclusive property of CIEPS and are protected under the provisions of the law of 1st July 1998 implementing in the Intellectual Property Code the European Directive of 11 March 1996 on the legal protection of databases. Any performance, whether total or partial, of this site by any company whatsoever, without the express authorization of the CIEPS is strictly forbidden and shall constitute an infringement sanctioned such as Intellectual Property Code.

(note, incidentally, that I'm already in conflict with section 3 of the legal notice: "Users and visitors cannot place a hyperlink to this website without the CIEPS' express and prior authorization."…)

The full database itself may be copyrighted, but (I believe) the individual facts within it shouldn't be. In which case, the best non-copyrighted source for the journal title, ISSN and ISSN-L of each serial is probably the publishers themselves, and as many publishers make their metadata through an OAI interface it may be possible to extract a fair amount of serials information from those sources. ScientificCommons, for example, harvests articles from OAI repositories, so may be able to aggregate useful title and ISSN information.

Existing, non-free sources

A commercial source of the information I'm looking to create is the JournalSeek database (from Genamics, licensed through OCLC) which includes around 100,000 journals.

SHERPA/RoMEO aggregates journal lists from Zetoc, DOAJ (7500 journals) and Entrez (40,000 journals), but the data is only available for non-commercial use.

Other sources of Journal/ISSN information

• 300,000 serials in Ulrich's (JSON interface somewhere?)
• SerialsSolutions Summon (title list as PDF)
• 13,000 journals in EBSCO "Academic Search Complete"
• 91,000 journals in WorldCat Local
• 17,000 journals in Thomson Reuters Master Journal List
• 27,000 journals in CrossRef
• 12,500 journals archived by Portico
• 9,000 journals participating in LOCKSS
• 3,330 journals in ScienceDirect
• Journal lists by subject in Bing Academic Search
• Search journals in the CAS Source Index (copyrighted by the ACS)

It would be nice if Freebase could serve as a central repository for this information, but there are only 4,321 journals in Freebase so far.

Abbreviations

Once we have the list of ISSNs and journal titles, we also need the corresponding journal title abbreviations, for use when generating bibliographies. There are lists of journal title abbreviations available for import into EndNote. Sadly, there are several different abbreviation styles (ISO, MEDLINE, BIOSIS, CASSI, etc).

• The ISSN International Centre maintains the list of Title Word Abbreviations which corresponds to the ISO 4 standard (available from the ISO store for ~£50), which describes the rules for abbreviating title words and titles of publications. The list of Title Word Abbreviations is available online as HTML.
• The NLM uses the list of Title Word Abbreviations to abbreviate periodical titles.
• The NCBI provides a less-comprehensive list of abbreviations for commonly-used English words in journal titles.
• The jabbrv LaTeX package abbreviates journal titles using the list of Title Word Abbreviations.
• The JAbbr service built a list of abbreviated serial titles from the Cornell library catalog of MARC records, and provides abbreviation → full title mapping as a JSON and HTML web service (source code provided). Article in Code4Lib Journal.
• The NLM Catalog is searchable using a journal title abbreviation, but uses sentence case for full titles.
• The NLM Catalog is also searchable by various ISSNs.
• Web of Science has a list of journal title abbreviations.
• A large list of Journal Abbreviation Sources.

More links

http://pinboard.in/u:hubpin/t:journals/

<!-- edit this; the PDF file must be on the same domain as this page -->
<iframe id="input" src="your-file.pdf"></iframe>

<!-- embed the pdftotext service as an iframe -->
<iframe id="processor" src="http://hubgit.github.com/2011/11/pdftotext/"></iframe>

<!-- a container for the output -->
<div id="output"></div>

<script>
var input = document.getElementById("input");
var processor = document.getElementById("processor");
var output = document.getElementById("output");

// listen for messages from the processor
window.addEventListener("message", function(event){
  if (event.source != processor.contentWindow) return;

  switch (event.data){
    // "ready" = the processor is ready, so fetch the PDF file
    case "ready":
      var xhr = new XMLHttpRequest;
      xhr.open('GET', input.getAttribute("src"), true);
      xhr.responseType = "arraybuffer";
      xhr.onload = function(event) {
        processor.contentWindow.postMessage(this.response, "*");
      };
      xhr.send();
    break;

    // anything else = the processor has returned the text of the PDF
    default:
      output.textContent = event.data.replace(/\s+/g, " ");
    break;
  }
}, true);
</script>

See an example running as a live demonstration.

It'll only work in recent browsers, as it requires sending binary data between windows as an ArrayBuffer using window.postMessage, and Web Workers in pdf.js.

Basically, this fetches a PDF as an ArrayBuffer using XMLHTTPRequest, then posts it to the embedded window, which uses pdf.js to render the PDF to Canvas (invisibly; you can see the rendered images if you poke around a bit with a web inspector tool). As it does so, an HTML layer is constructed, containing a block to match each row of the PDF - this would normally be overlaid on top of the rendered images to allow text to be selected, a technique used by many services that allow PDF text selection and highlighting, including Crocodoc and Google Docs' PDF viewer. By taking the text content of those blocks, the service can return the contents of the PDF as a single block of text.

I expect that pdf.js will acquire a native function for retrieving the text content directly, to make documents searchable. It would be nice, next, to try to recreate paragraphs by looking at the spacing between the blocks, and to use the formatting and other heuristics to extract metadata like title, authors, etc.

We now have a standard way of providing metadata about any object, based on two principles:

1. Every object is represented by at least one HTML page on the web.
2. Properties of that object are represented as <meta> elements in the <head> section of that HTML page.

From that, we can make statements about any object using URIs, and fetch metadata about that object using HTTP. The Semantic Web!

Statements

This is an RDF statement:

That's two things connected by a link, all represented by URIs.

Creating a Graph

Several of this kind of statement can be combined to make a graph:

Or, to write those statements in shorthand, without repeating the first part of each one:

And using prefixes to avoid having to write out the full URI each time:

Fetching information

The URI <http://music.com/band/nirvana> represents the band Nirvana. We could equally have used <http://en.wikipedia.org/wiki/Nirvana_(band)> or <http://open.spotify.com/artist/6olE6TJLqED3rqDCT0FyPh>*. As these are HTTP URLs, a representation of this Thing can be fetched using HTTP - in this case, your web browser probably receives an HTML representation of the band.

How does the server decide in which format to return that information? There's a negotiation between whoever requests the information and the server that provides the information. The request contains a list of formats that it would be able to handle, and the server returns the first of those that it's able to provide.  In fact, the information about a Thing might be available as JSON, or XML, or any other format, but Open Graph requires that every Thing identified by a URL must have an HTML web page that represents it.

In this way, we can make statements about any Thing, and fetch information about that Thing by dereferencing its URL to see what information it provides.

How should the information about the Thing be presented in that HTML page**? As the page represents the Thing***, this information can be added to the <head> section of the page; it ends up looking like this:

Which is exactly the same information as in the shorthand RDF statements above. It's RDF in HTML!

If someone says they like the album "Nevermind", a statement is created:
<http://facebook.com/eaton.alf>    <http://example.com/emotions/likes>    <http://open.spotify.com/album/6okv1avxEgYSdc2JYy6ZEi>

When we fetch the HTML document from the URL referenced (<http://open.spotify.com/album/6okv1avxEgYSdc2JYy6ZEi>), it contains (amongst other things) this information:

And when we fetch the HTML document from the "musician" URL <http://open.spotify.com/artist/6olE6TJLqED3rqDCT0FyPh>, it contains (amongst other things) this information:

When all that information is combined, we know that this person, who clicked the "Like" button while listening to an album in Spotify, liked the album "Nevermind" by the musician "Nirvana" - which is what gets displayed in their Facebook timeline.

Referring to URLs

This is all relevant to my recent post about citing with URIs. In that demonstration, the script dereferenced the URI to get information about the thing, but specifically asked for JSON. In the end, though, the JSON is basically just a list of properties about the thing being referenced, and there's no reason why that information can't be represented in <meta> elements in the <head> of an HTML page, which is exactly what most publishers do in order to get their documents indexed by Google Scholar. They use several prefixes ("dc.", "prism.", "citation_"); they often use meta[name][content] instead of meta[property][content], but it's all basically the same thing. I've now updated the script to parse <meta> elements from HTML, alongside JSON responses.

In summary: if someone wants to refer to a Thing, they should be able to use a HTTP URL. If someone wants to get information about that Thing, they should be able to dereference that URL, get an HTML document, look in the <meta> elements in the <head> section, and retrieve all the information about that thing (including further URLs to find out more information about any of those properties).

* Asserting equivalence between URIs allows links from one URI to also apply to the other. For example:
<http://example.com/music/nirvana> <http://www.w3.org/2002/07/owl#sameAs> <http://open.spotify.com/artist/6olE6TJLqED3rqDCT0FyPh>

** We don't have to worry about representing multiple items on a single page - each one will have a link to its own, individual page.

*** We don't have to worry about whether the URI represents the Thing or a document about the Thing: it's always the Thing. Most of the time, no-one cares who wrote the document about the Thing, or when that document was last updated. An exception might be Wikipedia, so I have a suggestion: the Thing is still represented by the web page; information about authors and update times can be attached to an appropriate property of the Thing, e.g. <http://en.wikipedia.org/wiki/Nirvana_(band)#description>.

Citing With URIs

The most straightforward way of being able to cite something in a document is to insert an identifier. On the web we hyperlink using URLs, which provide a unique identifier for the item being referenced - with the added bonus of being able to follow that URL to retrieve the item. When writing a scholarly article, however, there's still an expectation that the metadata for a citation will be provided, so that the reference will still make sense even if the URL stops working.

To be able to successfully cite using identifiers, therefore, means being able to retrieve the metadata for each identifier, and the simplest way to do that is to convert that identifier to a URL - if it isn't already - and retrieve it using an HTTP request.

Once we have the metadata for each citation, all that's needed is to generate a bibliography (a list of endnotes) at the end of the document, and insert links to those references inline. As a complication, there are many different publishing systems, and they each have their own special preferred formatting for those inline citations and bibliographies, so the tool should ideally be able to cater for any of those formats.

There is a need for citation software that works with Google Docs, as it's basically the standard online writing tool (and is continually getting more awesome). I've managed to get the first steps of a citation processor working in Google Docs; it's not complete yet...

Inserting and Processing Citations

Google Apps Script provides a way to add menu items to Google Docs and call a function when a menu item is selected. It's server-side Javascript, with an online editor that functions well. You can currently only attach scripts to Google Spreadsheets, but that's ok in this case: we need somewhere to store a local copy of our references.

Here's how to use Google Apps Script to format citations in a Google Document:

1. Create a new Document in Google Docs and give it a unique title.
2. Write your article, adding citations inline in the form {{cite:doi:10.1038/nchem.1108}}.
3. Create a new Spreadsheet in Google Docs and give it a title which is the same as the document, but with " - References" at the end.
4. Add my Exciting script to the spreadsheet (Tools > Script Editor). Once it's installed, an "Exciting" menu should appear.
5. From the "Exciting" menu, select "Generate Bibliography".

The script will now create a copy of the document (which must be in the same folder as the spreadsheet, and have the same name minus the " - References" suffix). The original document will remain untouched. It will parse the document for {{cite}} strings, fetch the metadata for each one, and store the data in the current spreadsheet (if the script is run a second time, it will use this local data instead of fetching it again). It will then replace the inline citations with numbered references, add a formatted bibliography at the end of the document, email you a PDF of the final, formatted document, and move the formatted copy of the document to the trash.

[NB: this is a first attempt, written last weekend. The citation formatting is very, very basic.]

There are several ways to cite using this system, and this is where it gets most interesting:

• You can use {{cite:doi:10.1038/nchem.1108}} to cite an item by DOI; in this case the data will be fetched from CrossRef.
• You can use {{cite:mendeley:123456}} to cite an item using its ID in your Mendeley library; in this case the data will be fetched from mendeley.com (this might not be working properly yet - I haven't tested it much. It uses OAuth authentication, so you need to register an application and get a key from the Mendeley Developers Portal. You also need to run the "authorizeMendeley" function from within the Script Editor, to authorize this application).
• You can use any URL, as long as it returns JSON when specified in HTTP Accept headers. For example, {{cite:http://dx.doi.org/10.1038/nchem.1108}} works just as well as the DOI example above.

Theoretically, you can cite any URL, and the script will retrieve the metadata from that URL and make use of it. In practice, not nearly as many URLs as I'd like perform content negotiation and return JSON instead of HTML from the same URL, and even when they do there's no standard format for the reference metadata (which is where RDF comes in, but there's no RDF parser in Google Apps Script; RDF triples as JSON would be an good intermediate). The current script has custom functions to normalise the data returned from CrossRef and Mendeley into a single, standard format for local use; adding other sources would probably require a custom parser for their metadata as well.

I'm not able to enter the Mendeley/PLoS API Binary Battle, but |'d be delighted if anyone who's interested was to take this code and make use of it. I see the next steps like this, possibly: 1) get citeproc-node running on a node.js server somewhere (Heroku or Joyent, maybe), and use that for formatting the references; 2) use the UI Services/GUI Builder in Google Apps Script to build an editing interface, for tidying up references once they've been retrieved; 3) add the ability to specify custom formatting for the inline citations, and to choose the citation format for the bibliography.

This means that anyone can now make client-side PubMed search interfaces, like this one.

Only the eSearch and eSummary methods have the Access-Control-Allow-Origin header so far, so it's not possible to get abstracts or full citation data this way (using eFetch) yet.

var url = 'http://www.guardian.co.uk/'
var output = 'snapshot.png'

switch (phantom.state.length){
    case 0:
        phantom.state = 'rasterize'
        phantom.viewportSize = { width: 1024, height: 768 }
        phantom.open(url)
    break;

    default:
        document.querySelectorAll("#guardian-logo img").item(0).setAttribute("src", "http://placekitten.com/344/52")
        phantom.sleep(1000)
        phantom.render(output)
        phantom.exit()
    break;
}

1. /
2. /?_limit=5
3. /?_limit=5&_start=20
4. /?tags=firefox&_limit=10
5. /?tags=firefox&_limit=10&_format=json
6. /?_format=atom

Item

1. /archives/001943.html
2. /archives/001943.json

(archive URL structure retained from the old version, for compatibility)

POST/GET/PUT/DELETE

At the recent "Beyond The PDF" conference in San Diego (which was pleasantly easy to attend remotely, because anyone could follow the webcast and participate via Twitter) there were several sessions that discussed entity extraction and manual annotation of academic papers. This, therefore, seems like a good time to write about the annotation system and user interface that I worked on at Nature Publishing Group over the last couple of years.

I'd been trying out annotation systems, and others had been working on similar things, for a while (the RSC's Project Prospect launched in 2007 and was being presented at conferences, while Phil Bourne presented several authoring tools at Data Webs, the first conference I attended after joining NPG). This project started out as a discussion and early prototype with Robert Hoffman, who had recently launched WikiGenes and had a biological entity extractor in use on iHOP; the initial aim was to allow authors to annotate genes and proteins mentioned in their papers. The focus of the project switched quickly to chemistry, though, as the launch of Nature Chemistry was getting close; from then on, this work was performed in close collaboration with the Nature Chemistry team, particularly technical editor Laura Croft, who provided ideas, most of the feature requests and user interface requirements for this system.

The annotation workflow expanded last year to cover more journals and now includes annotation of both chemical entities (compound names/molecular formulae) and biological entities (gene/protein names).

The system comprises five main parts:

1. Input: When an article XML file is uploaded, a specified list of elements (title, abstract, body, tables and figures) is converted to HTML using an XSL template. A configuration file specifies which of the original elements are blocks (which become divs in the HTML) and which are inline elements (which become spans). Some elements, such as links, get special treatment, and all element names are carried over to the class names of the HTML elements for styling. All named entities are converted to UTF-8 characters, and no characters are added to or removed from these elements, so the character positions in the HTML and the original XML are identical.

2. Entity extraction: The content is passed through several automatic entity extractors, each of which is specialised for a particular type of entity (chemical names, gene names, place names, etc). As most entity extractors prefer to parse plain text rather than HTML, the content is converted to text and separator characters are added to prevent annotation across the boundaries of block-level elements.

   When the results of the entity extraction are returned from each extraction web service, the annotations are converted to a standard format, which is then stored in MongoDB. By accounting for the previously-added separator characters, the positions of each annotation can be correctly translated back to positions in the HTML/XML.

3. Curation: The HTML is displayed in a web browser as several identical overlaid layers: one base layer containing no annotations, one layer for each set of automatically-extracted annotations and one layer for each set of manually-curated annotations. The display of each of these can be toggled on or off by the curator, allowing several sets of annotations to be displayed concurrently without breaking the DOM by overlapping elements.

   Each set of annotations is loaded on-demand, as JSON, so that the initial rendering of the page is fast. The text is transparent on all layers except the base layer, so it doesn't cause anti-aliasing artifacts, and the CSS pointer-events property is used to pass all clicks through to the base layer; highlighting a passage of text thus creates an annotation only in the base layer. Annotations in each layer are represented as inline spans: these have visible text, colours to show their state, and can receive clicks regardless of which layer they're in (the z-index of each layer determines which layer's annotations receive clicks in preference to other layers; the manual sets of annotations are in the foreground).

   Each annotation can have a single entity attached to it: a data object with a set of metadata properties appropriate for the type of entity/annotation being curated. Once an entity is chosen from the search results (see below) and attached, the annotation is copied out of the set of automatic annotations and into one of the manual sets: these are the annotations which are going to be published.

4. Search: Creating a new annotation, or clicking on an existing annotation (which selects all annotations of the same text in the current document), launches a search across several databases, chosen according to the type of annotation being curated (chemistry, biology, etc). The curator can choose which of the known properties of the currently attached entity to search on: the default is to run a search on the "title" property using the text of the annotation. The results from each search source are converted to a standard format (currently HTML with pseudo-RDFa markup rendered server-side, but could easily be JSON rendered client-side into a template), and the search results are displayed. When one of the search results is selected by the curator (from any of the search sources), the entity represented by that search result is attached to all of the annotations currently being edited, replacing any entity already attached. A list in the sidebar keeps track of all the attached entities in each annotation set.

5. Export: The positions of the curated annotations are spliced back into the article XML, which then re-enters the publishing workflow. The annotations themselves - including the entities attached to each annotation - are stored in an XML database for retrieval when the article is rendered as HTML, where they are matched back up to the annotation positions inline in the article XML; this storage also allows the annotations to be published independently via an OpenSearch/SRU gateway.

This project is ongoing: there is much work to be done on streamlining the user interface and adding more features for chemistry and biology curation.

From a technical point of view, there are several things which could be improved, including using a system like Backbone.js to separate the data model from the DOM (making it easier to synchronise changes between the annotation data, the front-end display and the server-side storage). It might, perhaps, turn out to be better to store annotation positions relative to each node, and give each node a unique ID using the XPath for that node (as PLoS use for their public annotations system), rather than the more fragile system used here which counts the distance of each annotation node in characters from the start of the document.

The key benefit of this system is that it's straightforward to plug in more automated entity extractors as they become available: by standardising the input and output formats, we can make use of as many entity extractors and search sources as possible. The automated annotations are mostly used as hints to the human curators, though, so being able to store the corrections that the curators make and feed those back to the entity extractors will be a big improvement - not many automatic annotation services are set up to learn from manual feedback, yet.

As more search sources are added to the system, the similarities between this and Paolo Ciccarese's Semantic/SWAN Annotation Framework become more and more obvious (Paolo's work inspired the use of annotation sets here, for example). In the SAF, each entity is selected from a set of ontologies rather than from a set of databases, but basically the process is quite similar.

We're storing the properties of each entity as XML using simple key/value pairs (using CURIEs as the keys) in MarkLogic, but when publishing these annotations I hope that they can be published using both the Annotation Ontology and OpenAnnotation ontologies, which have similar aims in standardising the representation and publication of annotations.

While I'd like to be able to open-source the code for anyone to use, it's probably going to remain locked up. As alternatives, there's some excellent work on an annotation system at the Open Knowledge Foundation (built for annotating in the Open Shakespeare project), the automatic markup of entities in PubMed Central UK (using the modular text-mining system Whatizit, developed and maintained by Dietrich Rebholz's group at the EBI), the Semantic Annotation Framework mentioned above (being applied to a similar purpose as our tool, for curating the results of text-mining services, in collaboration with Elsevier), OntoText's Linked Life Data platform (not specifically about annotation, but lots of text mining and linked data) and many others.

Test page.

View it on AOTY, or as a Spotify playlist.