As we come to the final month of the JISCPress project, we had some great news over on WriteToReply last week where we were able to announce that Eduserv would be covering our hosting costs for the immediate future (Eduserv funds hosting for WriteToReply, eFoundations: Write To Reply).

So what exactly does the platform we’ve been working on have to offer? Here’s one of the ways I think of it…

A document publishing platform that automatically atomises documents to the paragraph level, allows aggregated commenting at the paragraph and ‘user’ level, and supports the republication and re-presentation of documents in a variety of standard formats at the document level.

The first part of the process is the (manual assisted) ingress stage, in which documents are imported into the WordPress environment such that each substantive document section ideally maps onto a single WordPress “blog post”:

An RSS for the document as a whole, with one item per section, is generated automatically by the WordPress platform. A single item RSS feed is also generated for each page (so the content of each page can be easily transported around the web).

The second part of the process is the atomisation of each post, carried out automatically by the Digress.It theme, in which each paragraph in the document is given its own unique URI, derived from the URI of the web page (“blog post”) the paragraph appears on:

Potentially, an RSS feed can also be produced for each page in which each paragraph is a separate feed item, thus allowing a page/section to be transported around the web via a single feed, but in atomised form.

The paragraph level chunks produced by the atomistation process can be transcluded as independent elements in independent web documents in other documents by a variety of means (as an embeddable object, via XML, txt, JSON, etc):

The default nature of the WordPress platform allows comments to be made at the level of each web page, with an RSS feed of comments for each page being published ‘for free’. JISCPress extends this functionality by allowing comments to be associated with discrete paragraphs. Views over the comments are also available at the user level, (that is, grouped according to the user who made the comments, wheresoever they are made in the document). An additional RSS fed of comments by user is also available, which means that a document on the platform can actually be used as a scaffold for a critical response to the document by a particular user.

A further level of innovation is based on the automated generation of ’semantic tags’ at the page level. Once generated, tag based collections of posts can be syndicated in the normal way via WordPress generated tag based RSS feeds:

JISCPress also benefits from the Trackback mechanism implemented by WordPress. When a page or paragraph URI is linked to from a third party web page, a trackback to the originating page may be captured, which we interpret as the automated capture of links remote annotations or comments about the document.

When considered in these terms, the JISCPress/WriteToReply platform is seen to provide a powerful means of publishing documents in which individual sections may carry their own unique URI, and individual paragraphs within a section also contain their own unique URI (which in many situations may be rooted on the section URI).

The platform can also be regarded as republishing – or re-presenting – each section (i.e. page) and each paragraph as an independent entity. That is, whenever a document is published via the platform, each separate paragraph may also be thought of as being independently published “for free”, in the sense that:

- each paragraph is independently addressable,
- each paragraph is independently commentable, and
- each paragraph is independently republishable/syndicatable.

So, given that, can you think of any ways in which the JISCPress/WriteToReply platform can support your document publishing and comment gathering strategy?

I have been invited by the JISC-funded Edspace project, based at the University of Southampton to give a talk at an event on “Traditional educational repositories v. Web 2.0 resource sharing” to be held on Wednesday 4 November 2009. I have been asked speak on “the future for educational resources and services on the Web” – a rather grandiose topic, I think! I’ve entitled the talk “The Future for Educational Resource Repositories and Services in a Web 2.0 World” as its the Web 2.0 aspect I feel is important (and reflects my area of expertise – I don’t claim to have anything particularly significant to say on the repository side of things).

I’ll be saying that many of the technical aspects of Web 2.0 are now mainstream – and indeed the Edspace’s Edshare service provides RSS feeds, tag clouds, embed functionality and ‘cool URIs’.

But the term Web 2.0 also covers the network as the platform and a culture of openness. The issue of openness of educational resources is being addressed in, for example, the JISC OER programme and although I personally seek to ensure that my content (such as blog posts, slides and papers) are available under a Creative Commons licence I know that there are added complexities in the area of educational resources – so I’ll not focus on the openness issue.

Instead I’ll raise the question of the network as the platform in the context of the futures for educational resource repositories. I’ll suggest that as experts predict further cuts in the public sector, including higher education, wouldn’t it be appropriate for our repository services to be hosted in the cloud? And the concerns which tend to be raised (sustainability, reliability, legal issues, etc.) are implementation details which do need to be addressed – but these aren’t the important policy issues.

The slides I’ll be using are available on Slideshare (in the Cloud(!) although a master copy is also held locally) and is embedded below.

I received an email yesterday about the Scopus API, so thought I’d play with it for a bit of a ‘Friday afternoon experiment’. So here is a quick recipe for adding citation counts to DSpace’s JSPUI:

1. Register for the Scopus API service: http://searchapi.scopus.com/
2. Register your website (e.g. http://dspace.example.com/): https://searchapi.scopus.com/developerProfile.url
3. Download and save this patch (it only edits two files – display-item.jsp and header-default.jsp)
4. Edit the patch and insert your developer ID where you see XXXXXXXXXXX in the Javascript
5. Apply the patch to your DSpace instance
6. Re-build, and redeploy your DSpace instance
7. Visit any item that has a DOI stored in the dc.identifier.doi field
8. Look out for the citation count appearing at the top (if the item has a count of more than 0!)

Lewis, S., Hayes, L., Newton-Wade, V., Corfield, A., Davis, R., Donohue, T., Wilson, S., If SWORD is the answer, what is the question?: Use of the Simple Web-service Offering Repository Deposit protocol, Program: electronic library and information systems, 2009, Vol 43, Issue 4, pp: 407 – 418, 10.1108/00330330910998057, Emerald Group Publishing Limited

Of course a copy is available open access in our repository: http://hdl.handle.net/2292/5315

Abstract:

Purpose – The purpose of this paper is to describe the repository deposit protocol, Simple Web-service Offering Repository Deposit (SWORD), its development iteration, and some of its potential use cases. In addition, seven case studies of institutional use of SWORD are provided.

Design/methodology/approach – The paper describes the recent development cycle of the SWORD standard, with issues being identified and overcome with a subsequent version. Use cases and case studies of the new standard in action are included to demonstrate the wide range of practical uses of the SWORD standard.

Findings – SWORD has many potential use cases and has quickly become the de facto standard for depositing items into repositories. By making use of a widely-supported interoperable standard, tools can be created that start to overcome some of the problems of gathering content for deposit into institutional repositories. They can do this by changing the submission process from a “one-size-fits-all” solution, as provided by the repository’s own user interface, to customised solutions for different users.

Originality/value – Many of the case studies described in this paper are new and unpublished, and describe methods of creating novel interoperable tools for depositing items into repositories. The description of SWORD version 1.3 and its development give an insight into the processes involved with the development of a new standard.

The seven case studies include a thesis submission system, a SWORD plugin for moodle, an automated laboratory data repository deposit tool, a desktop deposit tool, the BibApp repository integration module, a custom deposit tool for a technical report series, and the Facebook SWORD deposit tool.

If you have not yet taken a look at CDL's curation micro-service specifications, most of which may be printed on as few as one or two sheets of paper, see the Digital Library Building Blocks.

My co-workers in the LC Repository Development Center have been chatting about these specs on and off throughout the year. After months of procrastinating, I finally read all of the specs on Thursday; it's wonderful that you can do so in the course of one reading session, I might add. Yesterday a bunch of us RDCers got together to chat (informally) about the specs: what they're for, how they work, and how they interact with one another. I learn by doing, by examples, so I combed through each of the specs in advance of our meeting and tried to construct a minimal repository[1] based on micro-services.

Here is a tree visualization of the final product, inevitable warts and all: The services I used were Namaste, Content Access Node (CAN), Pairtree, Dflat, Reverse Directory Deltas (ReDD), Class-based System for Managing Object Properties (CLOP), and BagIt (co-developed by LC and CDL).

As I mentioned in our Friday meeting, recounting my experience exploring the specs: the bad thing is that I spent an hour building a repository with rudimentary tools such as mkdir, touch, cp, ln, and emacs; but the good thing is that I built a repository in one hour using common, rudimentary tools. It's a very compelling paradigm. Ed's already built a tool implementing some of Dflat, further demonstrating how lightweight these micro-services are. (UPDATE: Ed notes that this code is a work in progress and is "barely functional.") (UPDATE 2: The dflat library has come a long way. Check it out if you're interested. Also, I just committed a pretty basic Namaste library: http://github.com/mjgiarlo/namaste. Only took about an hour, which is a testament to the power of lightweight specs.)

I am certain this will be a running thread at work as the specifications evolve and our understanding of them grows. Some questions and comments that occurred to me while exploring the micro-service specs and building the minimal repo:

• CAN was a bit puzzling. The spec is simple enough, but I found some of the conventions confusing, and I was left wondering what CAN provides other than a container. What I would like to see is a simple use case and perhaps more examples. Thus, the CAN stuff in my sample repo doesn't feel very useful only because I had a hard time working with the spec.
• CLOP feels like the least mature of the specifications. It seems generally useful to be able to put digital objects, however you define that, into classes and define properties on those classes. The spec did not clearly convey to me just how it accomplishes that aim. A few examples would go a very long way. I've got some CLOP stuff in the sample repo but I have no idea how close my implementation matches the spec.
• Is Dflat dependent on ReDD? One would assume not since there's an optional property in the dflat-info.txt file for specifying a delta scheme. But, say, could you stub out the v001 directory (reserved to hold the initial version of a digital object) and use a system such as git or bazaar?
  
  One might argue that these established delta schemes, if you want to call them that, have many more developers and users than a system such as ReDD and thus should persist longer and have more tools built around them. I imagine the micro-service viewpoint would acknowledge that point, but counter that the spirit of these specs is to avoid dependencies from outside the filesystem?
• Is the ReDD specification meaningful outside of a Dflat given that any one ReDD directory knows nothing of its successors and predecessors, or is it dependent upon Dflat?
• Could a BagIt bag live inside of the ReDD reserved "full" directory? That is, could the "full" directory be marked up appropriately to be a BagIt bag?
• How many tools exist for these specs? I notice there's code in CPAN for Pairtree and Namaste, which is a fabulous start. Tools are the difference between YAMF (Yet Another Messy Filesystem) and reliably managed curation services. Granted, tools such as cp and emacs already exist and are part of the appeal of these micro-services, but there's also tremendous room for error if operations are all done "by hand."
• To what extent has CDL transitioned to using these specs/tools?
• Are other institutions using these specs/tools? I have heard tell that digital library folks from the University of Michigan and the University of North Texas may be involved.

I hope I don't sound overly critical. I'm really glad our colleagues at the California Digital Library have written these specifications and applied their deep experience to what could be a transformative paradigm[2] in the digital curation world. Kudos to them!

Notes

1. Perhaps it's more in line with the specs to refer to this space as "a managed filesystem that drives repository and curation services," given the CDL philosophy that preservation is not a place/repository. But it's easier to say "repository," so there you go.
2. Please excuse the fanboyishness; this filesystem fetishism is exciting stuff!

The survey closed in July, and the subgroup spent August writing a report on the survey results. That report is now final and it's available to the public. Feedback may be sent to our (woefully underutilized) public i2info mailing list, left as a comment on this post, or e-mailed to me privately which I can forward to our internal list.

The next step is to build upon the report to draw yet more conclusions from the data — there's an awful lot there — and flesh out some repository use cases for institutional identifiers. The I2 core group is moving quickly towards finalizing identifier metadata elements so that a standard may be drafted, and I think having some use cases documented will help drive the standard in a direction the community can get behind.

Onward and upward.

However for developers working on DSpace, it can be a bit harder. For good reasons (e.g. spam reduction etc), many institutions make their SMTP email servers accessible only from within their network. So if you are working on DSpace from home, a cafe, or on the train, you don’t have access to your institutional network because you don’t want to bother running VPN. An ideal solution to this problem would be to use a third party SMTP server outside of your institution which you can use which developing. An ideal candidate for this is Google’s GMail.

At present though, DSpace does not allow you to configure some of the more complex aspects of email server configuration like enabling SSL connections, or which SocketFactory to use. These need to be set in order to connect securely to the GMail servers. We’ve had an open issue in the DSpace JIRA issue tracking system to address this, and it has now been included ready for DSpace version 1.6, and can be easily back ported to earlier versions.

To use the GMail servers, configure dspace.cfg as follows:

# SMTP mail server
mail.server=smtp.gmail.com # SMTP mail server authentication username and password (if required)
mail.server.username = your-user-name@gmail.com
mail.server.password = your-gmail-password # Pass extra settings to the Java mail library. Comma separated, equals sign between
# the key and the value.
mail.extraproperties = mail.smtp.socketFactory.port=465, \ mail.smtp.socketFactory.class=javax.net.ssl.SSLSocketFactory, \ mail.smtp.socketFactory.fallback=false

If you are a Google Apps user this will also work if you substitute the username with your Google Apps email address.

With this event, we decided to address an issue that has become apparent as JISC has started to engage more directly with developers in the HE sector: developers are often untrained, and sometimes not naturally disposed to explaining their projects to others, especially when those others are not themselves developers. So we hit on the (admittedly somewhat artificial) exercise of requiring a representative of each project to deliver a 45 second ‘pitch’ to the assembled audience, which was recorded to video. The project reps were then given a 15 minute consultation with one of a set of panels of three ‘experts’, led by people from media and communications backgrounds, where they reviewed the video of their pitch and discussed ways of improving upon it. These were held in the Stadium’s executive boxes overlooking the pitch, which was pretty cool! The reps were warned that they would be required to deliver a new, improved 20 second pitch the following day….

This exercise was something of a gamble to be honest. We were confident that a significant number of the project reps would appreciate why it is important that they be able to clearly explain what their project is about in a few sentences, in terms that a wide variety of people might be able to understand. We hoped that the majority would be able to benefit from these exercises to the point where they could deliver a compelling, clear pitch for their project. The results, I’m really happy to say, were outstanding! The improvement over 24 hours was remarkable, and JISC now has a portfolio of clear explanations of the 40+ ‘rapid innovation’ projects, not to mention a group of developers better equipped to explain what it is they are working on.

As well as a training exercise, this event delivered a series of ‘lightening talks’, panel sessions and ‘show & tell’ opportunities – a set of features which has become a staple of developer-centric events. Twitter was actively used as a back-channel to the event so you can get a small sense of what was going on from that stream.

I also used this event to ‘officially’ launch the DevCSI project – but managed to cock up my presentation – by ‘losing’ my presenter’s display with all my notes. As I’d decided to go for a one word per slide approach for much of the presentation, this was a bit of a disaster for me. Oh well, I gathered some real interest in this nonetheless and some opportunities for events and other engagements. If you’re interested, you can read more about this on the project blog.

I’d like to extend a big thank-you to everyone who came, many of whom stepped some way out of their ‘comfort zone’ to engage with the ‘pitching’ exercise. I’d especially like to thank Mahendra Mahey (UKOLN) who did most of the organising together with David Flanders (JISC), as well as the ever-professional UKOLN Events Team (Natasha Bishop and Michelle Smith) who seemed to work non-stop for 24 hours. Our army of professional bloggers was fantastic, offering expert advice on the pitches as well as conducting interviews, all of which have already been transcribed to the IE Demonstrator Blog, with a large number of the projects. David Tarrant (Southampton University) and Julian Cheal (UKOLN) provided excellent technical support, maintaining a networked news service which was displayed all over the venue.

This event was a pleasure to be involved in – there was a great spirit of cooperation throughout which bodes really well for future events with the developer community.

I’ve given this talk a couple of times, at the JISC Digtital Content Conference, The JISC INF11 Programme Start-Up Meeting and most recently (where I presented it virtually over twitter) at IWMW2009. It orgins stem back to chats with Ben O’Steen while on the CRIGshow and most recently in more serious discussions with OSSWatch in their “Agile and Open Expert Group“. I still have a lot of questions on how project jump from being small one or two user acorns into massive oak services with lots of users, if you have solutions please do comment or send me a tweet @dfflanders.