Nowadays, of course, there are many more communications channels available, and many users (in the case of events, potential participants) will expect to receive information of relevance to them in their preferred environment. Indeed, to use the visitors and residents metaphor, if they are ‘residents’ of the online environment they will expect the modern equivalent of the town cryer to make the announcements close to their residence, whereas ‘visitors’ may well expect to receive information only if they track down relevant information kiosks.

In addition to using tweets, blog posts, RSS feeds and LinkedIn announcements it is now possible to use video sharing tools, such as YouTube. Such popular services, which will be readily available on mobile devices, may be particularly useful in reaching out to people on the move, who may find it easier to view a brief video clip rather than read text on a small screen.

For this reason I have created a brief video clip, lasting just over 2 minutes, which summarises the IWMW 2013 event, which will be held at the University of Bath on 26-28 June. The video clip is available on YouTube and is embedded below. I should add that the questions were asked by Kirsty Pitkin and the video was taken by Rich Pitkin, who also edited the video. Kirsty and Rich are running a session on Creating a Multimedia CV or Project Summary at the IWMW 2013 event, so if you would like a brief video made about yourself or some aspect of your work, feel free to sign up for the session! Remember that the 3-day event costs only £350, which includes 2 nights’ accommodation.

View Twitter conversation from: [Topsy] | View Twitter statistics from: [TweetReach] – [Bit.ly]

This page on my blog is intended to act as a repository for rants. I’ll be the first to admit that these rants (by their very nature) are *not* well thought out and intended more as a place for me to vent about what frustrates me. The point in posting these rants is that at some stage (once I am a bit calmer), I can come back to view these rants in a more methodical and mindful way. The value in these rants is therefore not in the intellectual ideas, but rather in the emotional ideas that provoked their ‘knee jerk’ reaction.  I hope to learn from these rants by posting them publicly, and  (hopefully) through my friends and colleagues I can separate the emotional from my ideological responses.

Below in the comments you’ll find these rants, to receive these rants you’ll need to leave a comment and then tick the “Notify me of follow-up comments via email” box.

NB The other reason for having this page on my blog, is my perception that twitter is increasingly becoming a failed space for having short topical debates that often get kicked off on twitter and usually erode after three or more people join in the discussion.

Increase of international student tuition fees (vs national student tuition) in the UK per year and student enrollment.

What strikes me as a bit awkward in the coming scholarly revolution is how easy it is to manipulate data in the visual. Will we not see researchers employing similar data storytelling skills (in the form of infographics like the above ‘bubbles’ atop a bar chart?) for the purpose of getting more people to click, link and download their paper based on what they see on the first page (abstract + image)?

NB I’d highly recommend you read the JISC/OCLC report on Gen Y researcher reading habits: “Bouncing, Chunking and Squirrelling”. To grossly paraphrase this report: the researcher who puts a map or infographic on the first page of their research article, significantly improves their chance of their research being ‘squirrelled’ away by readers, which leads to more citations, which leads to increased researcher reputation.

My question to you is: do these types of hybrid-(info)graphs undermine the scholarly process? Perhaps even employing marketing techniques to the data so it is more appealing to the reader?

On the other side of the coin, is this article where journalists are feeling they are losing their creative storytelling abilities because of ‘data journalism’: http://techcrunch.com/2013/03/31/data-is-not-killing-creativity-its-just-changing-how-we-tell-stories/

Some other attempts at various infographics of the same data as the chart at the top of this post:

This one (as opposed to the one at the top of this post) doesn’t use logarithmic data but does place the trends in proportion to one another…

As Martin Hamilton pointed out in his post on A Tale of Two Jiscs: Reflections on CETIS13, FutureLearn and the JISC Diaspora ”In many cases, JISC was farsighted enough to forsee requirements in the research and education sector that have subsequently turned into significant businesses in themselves“. But Martin then went on to describe how those benefits are about to be lost: “we are entering a new era, necessitated by funding reductions, changing student demographics and frankly an unwillingness to see “R&D” type activities (of which a large proportion can be expected to fail) facilitated through top sliced central funding“. For myself and many of my colleagues we are having to respond to the scenario depicted by Martin:”Behind the scenes, a lot of people who have been working for JISC on its various centres and services have been having meetings with their local HR departments about redundancy and redeployment“.

But what should you do if you wish to continue to make use of the skills and expertise you have developed over the years but new full-time posts appear to be in short supply? I suspect the changes in Jisc will provide new consultancy opportunities, with their current preoccupation in telling good news stories without addressing any of the underlying complexities or tensions leaving a void which can be filled by those who have a more realistic understanding of the complexities of exploiting IT to support institutional requirements.

The preparation for a new career will mean the loss of an IT infrastructure and the accompanying support which many of us will have grown accustomed to. But how can provide help and advice in the preparation for a move away from an institutional environment? One might expect the Library to provide support, especially for institutions which have a commitment to information literacy, which is defined as “the ability to find, use, evaluate and communicate information” and is “an essential skill in this digital age and era of life-long learning“. But as I will be describing next week at the LILAC 2013 conference this is not necessarily the case, with the role of librarians perhaps being to promote use of institutional rather than Cloud services. But since we will all, at some point, leave our host institution, this is not really providing staff and researchers with the life-long skills needed to thrive beyond an institutional context.

Surely it is timely for a change in focus, especially if the gloomy predictions are correct and we continue to see reductions in staffing levels in higher education institutions?

I’d welcome your thoughts and comments – especially if you have experience of leaving your host institution and continuing to work, perhaps as a consultant. My slides are available on Slideshare and embedded below:

View Twitter conversation from: [Topsy] | View Twitter statistics from: [TweetReach] – [Bit.ly]

Whilst preparing a follow-up post on institutional repositories I started to explore the data which has been collected by the JISC-funded CORE project. The CORE (COnnecting REpositories) project aims to “facilitate free access to scholarly publications distributed across many systems“. The CORE Web site, which was developed at the Open University, provides access to four applications including:

Repository Analytics - A tool that enables to monitor the ingestion of metadata and content from repositories and provides a wide range of statistics.

I wanted to use this service to find information about the repositories provided by the 24 Russell Group universities. However, as can be seen from the accompanying screenshot, it was not easy to associate a repository with its host institution.

The first four examples illustrate the difficulties I had in using the information. The first entry, for the Aberdeen University Research Archive, gives a clear indication of the host institution. The second example, Abertay Research Collections, is somewhat more obscure, unless you know that Abertay is the name of a Scottish university. However the next two examples, Access to Research Resources for Teachers and Advanced Knowledge Technologies EPrints Archive, give no clue as to the host institution.

This meant that browsing the list was not an effective way of finding the repositories for the Russell Group universities. In addition the search interface was misleading: a search for “Southampton” enabled me to find eCrystals – Southampton and Electronics & Computer Science EPrints Service – University of Southampton - but not the main repository which has the name e-Prints Soton.

Using CORE to Search for Russell Group University Repositories

Despite the limitations caused by the lack of institutional identifiers I felt it would be useful to discover information held about Russell Group university repositories, based on a search of the CORE system using the obvious name for the host institution. The following table summarises the findings for a survey carried out on 21 February 2013 using the search term given in the second column.

Note that the Repository Analytics page does not appear to provide a formal definition of the data collected. However from hovering over the accompanying icon for the entries it appears that the Metadata Download column gives the number of metadata records, the Metadata Readable column gives the number of links extracted from the metadata and the PDF Download column the number of PDFs which were downloaded.

Discussion

It is difficult to interpret the data given in the table: the entry for the UCL Discovery repository, for example, tells us that there are 0 metadata records, with 245407 links having been extracted from these records and 2 PDFs downloaded!

However the table does suggest patterns of naming conventions for institutional repositories, such as the institutional name being provided at the beginning (“University of Birmingham Research Archive, E-prints Repository“, “University of Liverpool Research Archive” and “LSE Research Online”) or end of the repository name (“EPrints at the Centre for Scientific Computing, University of Warwick“, “Electronics & Computer Science EPrints Service - University of Southampton” and “Computer Laboratory Technical Reports - Cambridge University“) together with a large number of examples which use a partial form of the institution’s name (e.g. “Edinburgh Research Archive”, “Glasgow DSpace Service” and “Manchester eScholar Services“).

But of greater interest are the institutional repositories which have been harvested by CORE but are missing from this search such as “e-Prints Soton” and the “White Rose E-theses Online” and “White Rose Research Online” repositories which are used by the universities of Leeds, York and Sheffield.

Whilst the ownership of a repository will be apparent to the end user who access the service via the main entry point (perhaps from the institution’s Library Web site) in a number of cases such information is not apparent when the repository has been harvested and accessed using other systems such as, in this case, the interface developed by the CORE project.

In light of the findings from a survey of Russell group Universities, I would make the following simple recommendation:

Institutional repositories should contain the name of the host institution.

In order to illustrate the need for such a recommendation, here are a list of repositories which have been harvested by CORE:

Access to Research Resources for Teachers - Department of Computer Science E-Repository - Enlighten - Modern Languages Publications Archive - Online Publications Store - Open Research Online - Pharmacy Eprints

If you are unfamiliar with these repositories, would you to able to guess who owns them?

Or, to put it another way, meaningful metadata is important for repositories!

View Twitter conversation from: [Topsy] | View Twitter statistics from: [TweetReach] – [Bit.ly]

How significant is use of third-party repository services?

In a recent post I explained Why I’m Evaluating ResearchGate. In the post I summarised the reasons why I felt that researchgate.net could provide an additional service for depositing research papers which would complement Opus, the University of Bath institutional repository. But what others services might also be relevant? And which services are hosting the largest numbers of research papers?

In order to seek answers to these questions, I used Google to provide a measure of the size of a number of hosting services for PDFs and the number of PDFs they host. The services I analysed were:

• Researchgate.net: This site is described in Wikipedia as “a social networking site for scientists and researchers to share papers, ask and answer questions, and find collaborators. The site has been described as a mash-up of “Facebook, Twitter and LinkedIn” that includes “profile pages, comments, groups, job listings, and ‘like’ and ‘follow’ buttons”. Members are encouraged to share raw data and failed experiment results as well as successes, in order to avoid repeating their peers’ scientific research mistakes.“
• Academia.edu: This site is described in Wikipedia as “a platform for academics to share research papers. It was launched in September 2008. Currently the site is approaching 2 million registered users.^[2] The platform can be used to share papers, monitor their impact, and follow the research in a particular field.“
• Mendeley.com: Thus site is described in Wikipedia as “a desktop and web program for managing and sharing research papers,^[2] discovering research data and collaborating online. It combines Mendeley Desktop, a PDF and reference management application (available for Windows, Mac and Linux) with Mendeley Web, an online social network for researchers.^[3][4][5] Mendeley requires the user to store all basic citation data on its servers – storing copies of documents is at the user’s discretion“.
• Citeulike.org: This site is described in Wikipedia as “based on the principle of social bookmarking [the service] is aimed to promote and to develop the sharing of scientific references amongst researchers. In the same way that it is possible to catalog web pages (with Furl and del.icio.us) or photographs (with Flickr), scientists can share information on academic papers with specific tools (like CiteULike) developed for that purpose“.
• Scribd.com: This site is described in Wikipedia as “a document-sharing website that allows users to post documents of various formats, and embed them into a web page using its iPaper format“.

Many researchers will probably be familiar with the first four services listed. The fifth service, scribd.com, is included in order to explore whether a general-purpose PDF repository service could have a role to play in supporting the sharing of research publications.

Findings for the Coverage of the Services

Google was used in order to provide an estimate of the coverage of the services, including the total number of resources which have been indexed by Google and the number of PDF files. The findings are given in the following table. Note that the figures were initially collected on 6 February 2013. In order to check the volatility of the findings the searches were repeated on 11 February.

It seems that Scribd hosts a very large number of resources (although a finding of 3 PDF resources originally found was discarded as the results seemed to be unreliable).

However since Scribd is a general purpose repository service, it was felt that ResearchGate provides a repository of a large number of PDFs resources which are more relevant for researchers. In light of this confirmation of the popularity of Researchgate an additional survey was carried out which reported on use of the service across Russell Group universities.

Findings for Institutional Use of Academic.edu and Researchgate

On 1 August 2012 a Survey of Use of Researcher Profiling Services Across the 24 Russell Group Universities was published on this blog. This survey has been repeated in order to detect changes in the use of ResearchGate. Since the original survey also provided an analysis of Academia.edu, this was also included in the current survey. The results are given in the following table. Note that the data is also available in Google Spreadsheets.

Note: *  As described in the previous survey the numbers of Academia.edu members is obtained by entering the name of the institution in the search box.

Conclusions

Nos. of items deposited in Researchgate in Aug 2012 (blue) & Feb 2013 (red)

Nos. of Researchgate Members in Aug 2012 (blue) & Feb 2013 (red)

As illustrated in the accompanying diagrams it seems that the numbers of researchers who have signed up for a ResearchGate account has grown significantly over the past six months, and now stands at over 33,000 users, a growth of 82.7%. The numbers of papers which have been deposited by researchers at Russell Group universities has also grown to a total of over 477, 000 items. However since this represents a growth of 11.9% over six months it suggests that new members are providing metadata records only and not depositing the full text.

I therefore conclude that the conclusions I reached in my post which explained Why I’m Evaluating ResearchGate were correct and ResearchGate is a service which I should use not only to provide a presence about my research activities but also to host my research papers. I do wonder, though, whether the large numbers of items which have been deposited in ResearchGate is due to promotion of the service with the Russell Group universities or represents a bottom-up approach, in which researchers have recognised the benefits of the service and recommended it to their peers?

View Twitter conversation from: [Topsy] | View Twitter statistics from: [TweetReach] – [Bit.ly]

In a recent post which explained Why I’m Now Embedding ORCID Metadata in PDFs I described my intentions to ensure that my research papers contains rich embedded metadata to held enhance the discoverability of the publications, ensure that authorship is asserted (by embedding the ORCID ID of the authors of the papers) and ensure that embedded images contain descriptions which help ensure that the content can be understood by visually impaired readers. In addition I wish to ensure that the PDF is stored in PDF/A format which provides a more preservable format.

In light of discussions on the blog and on email I have decided to embed the ORCID IDs for co-authors of my peer-reviewed papers although, as suggested by Geoffery Bilder, I will be embedding the HTTP URI version of the ORCID IDs (e.g. http://orcid.org/0000-0001-5875-8744) rather than just the ORCID ID itself (0000-0001-5875-8744). In addition I will also be embedding the DOI for papers which have been assigned a DOI.

But I am now faced with the problem of where the paper should be hosted. This post summarises the processes I am using in the selection of an appropriate repository service to complement my institutional repository.

Selection Processes

As described previously workflow processes used in the creation of cover sheets for items hosted in our repository means that metadata embedded in PDFs is lost. Although we’re having discussions with repository staff about this, it occurred to me that I now have an ideal opportunity to make use of a third-party repository service.

In the past I have normally deposited papers in my institutional repository and used third-party services (such as ResearchGate and academia.edu) to host the metadata, with links being provided to the full-text of the papers hosted in the institutional repository. The main reason for doing this was to ensure that usage statistics for accesses of the full-text was available in a single location rather than being fragmented across a range of services. There was a need to minimise the effort in collating such statistics for the product of evidence reports of our work which our funders have required in the past. However in light of the recent announcement of the cessation of core-funding for UKOLN, this is no longer a priority! Indeed it is now important to ensure that ideas described in peer-reviewed papers are widely disseminated.

Using ResearchGate

Having recognised the value of hosting PDF copies of my papers on a third-party repository service the question then was which one to select. The key criteria used in the selection were:

• Easy to upload files.
• Popular with readers.
• Resource is easily found using Google.
• PDF files preserved intact.
• Service appears to be viable.

On 25 December 2012 I received an automated email from ResearchGate which informed me that “28 of your colleagues from University of Bath have joined ResearchGate in the last month“. On 24 January 2013 an automated message announced “44 of your colleagues recently joined ResearchGate“. As illustrated the University of Bath”s entry of ResearchGate shows that there are currently researchers from 26 departments who have uploaded a total of 7,263 publications. It seems ResearchGate is growing in popularity, at least at the University of Bath.

On 20 December 2012 I was notified of the numbers of views of my papers (or, more accurately, the numbers of views of the metadata for my papers): “Your published research was viewed 1,678 times in 2012” so perhaps ResearchGate is popular beyond the University of Bath!

In light of the apparent popularity of the service I decided to upload one of my papers to the service: the PDF copy of the paper on “Developing A Holistic Approach For E-Learning Accessibility“.

It was trivial to upload the paper, especially as the associated metadata had been created previously. I then downloaded the PDF and was able to confirm that the metadata was still embedded in the PDF resource.

The paper can be accessed from ResearchGate and the user interface is shown below. I’ll leave others to judge the usability of the service.

Page on ResearchGate for one of my papers

But in addition to users who are linked directly to the paper or access resources on the ResearchGate service using the Web site’s browse and search functionality, what of the discoverability of resources using Google.

ResearchGate, Google and Embedded Metadata

The PDF version of the paper now contains content which will not be widely used elsewhere: a combination of the authors’ names and their ORCID ID. A Google search for “Brian Kelly ORCID: 0000-0001-5875-8744“, “Lawrie Phipps ORCID: 0000-0002-0834-273X” or Elaine Swift ORCID: 0000-0002-6101-6861“ should initially find information about the paper hosted on the UKOLN Web site, the UK Web Focus blog and other services which may be used by the co-authors, although not the institutional repository as this does not currently provide ORCID information (understandably, as ORCID is so new).

I have therefore provided links to the following Google searches which I will monitor to see when Google has indexed the PDFs hosted on ResearchGate:

It appears that over a period of a week the ORCID metadata is being found from citation records hosted on the UKOLN Web site together with the citation records already indexed on the ORCID Web site and this blog, but not yet the PDF files hosted on ResearchGate. Might this be due to Google not indexing the researchgate.net site? In order to answer this question Google was used to provide information on the total number of resources on the service and the total number of PDF files. The results are given below.

* The numbers of search results have fluctuated from 24,100,000 – 55,300,000 during the last few days.

It seems that a large number of PDF files hosted on Researchgate have been indexed by Google, but it takes longer than a week for new resources to be indexed and the results found using a Google search.

Sustainability of the Service

What Does The Evidence Say?

The home page for the service displays a graphic (to users who are not logged in) of the numbers of the service. It seems that 2.4 million users have subscribed. Since there are likely to be researchers, this does appear to be a significant number.

But what else do we know about the service and the company which provides the service? TechCrunch provides a handful of posts about the company together with the following summary:

ResearchGate is the leading social network for scientists. It offers tools and applications for researchers to interact and collaborate. ResearchGate offers a social, crowdsourced platform designed for researchers. The platform provides a global scientific web-based environment in which scientists can interact, exchange knowledge and collaborate with researchers of different fields.

The results of ResearchGate’s new search engine, called ReFind, are not merely based on keywords, but selected in an intelligent way based on semantic, contextual correlations.

In addition the article also provides a graph showing the numbers of users over the past year, based on figures provided by Compete.

As can be seen, the numbers of unique visitors seem to be growing significantly, from 61,640K in December 2011 to 236,170K in December 2012.

I also used MajesticSEO to report on the SEO characteristics of the service (note free subscription required in order to view findings). As can be seen there are 7,459 domains which have links to Researchgate.net and a total of 177,945 backlinks. Although such figures need to be regarded with caution (for example, they can be skewed significantly by link spam) the number of links from educational domains (3,241) and the numbers of educational domains (551) may be more appropriate to measure, due to the difficulties in creating educations domains to host link farms. This snapshot may therefore provide a useful baseline for measuring changes in the link popularity in the service.

Terms and Conditions

It should be noted that looking at the ResearchGate terms and conditions I found no suggestions that the company claims rights to sell my data or my attention data to others (although I haven’t studied the terms and conditions in great detail). Although some may welcome this, others may wonder what the business model for the company is. An article entitled ResearchGate Wants To Be Facebook For Scientists published by Forbes in March 2012 described how:

“ResearchGate will also be looking into ways to monetize its platform. The “no-brainer” way to do that, in Madisch’s words, is to provide job boards for scientists looking for jobs. Universities and companies would pay the site to place listings. The company is also looking for ways to partner with other companies that manufacture and sell biotech lab equipment, as well as several other different programs.”

 Perhaps this is an appropriate business model which will accepted by researchers who normally shy away from free services on the grounds that “If You’re Not Paying for It; You’re the Product“.

Interest in UK HE Sector

Although ResearchGate seems to be growing in popularity globally (and in the University of Bath) is there any evidence of interest with the UK’s higher education community? For me this is not necessarily a significant issue (it can be fine to be an early adopter) but it would be interesting to see what others in my community are saying about the service.

Using a Google search for “researchgate terms and conditions“ I found that the DCC have provided a summary of ResearchGate in its list of resources of digital curators with a similar resource being provided by the University of Edinburgh’s College of Humanities and Social Science. A Google search for “researchgate UK“ finds a number of additional resources from the sector including pages provided by the University of Leeds (PDF format), the University of Leicester, the University of Liverpool (PDF format) and the University of Gloucester together with blog posts at the University of Loughborough and the University of Warwick.

My Decision

In light of these figures and my experiences in using the service I am happy to use the service to provide additional exposure to my research papers which complements the master copy of papers which are hosted on my institutional repository. Are other researchers making similar decisions or are alternative services felt to provide better options?

View Twitter conversation from: [Topsy] | View Twitter statistics from: [TweetReach] – [Bit.ly]

The draft will be available for public comment until March 15th 2013 – you are invited to comment via the ResourceSync Google Group. Group discussions are openly accessible; posting requires group membership.

In Herbert’s words:

The ResourceSync specification describes a synchronisation framework for the web that consists of various capabilities that allow third party systems to remain synchronised with a server’s evolving resources. The capabilities may be combined in a modular manner to meet local or community requirements. The specification also describes how a server can advertise the synchronisation capabilities it supports and how third party systems can discover this information. The document formats used in the synchronisation framework are based on the widely adopted Sitemap protocol.

ResourceSync is a collaboration between the National Information Standardization Organization (NISO) and the Open Archives Initiative (OAI). It is funded by the Alfred P. Sloan Foundation and Jisc.

This report, which provides a close look at three institutions (UCL, Imperial War Museums and the National Library of Wales) in the United Kingdom confirms:

• How fragmented the digital landscape is at universities and within other organisations.
• How there are examples of good practice within and outside higher education that all can learn from but that greater co-ordination is required to deliver this at a UK level.
• How little the topic of post-build sustainability comes up at the higher levels of administration.
• How risk is present within the current system, concerning the sustainability of digital content.

The report (which is available in PDF format) is substantial, containing 88 pages. In addition to this main report a second document (also available in PDF format) provides a “Sustainability Health Check Tool for Digital Content Projects“.

This report is very timely arriving at a time in which we are seeing reductions in the levels of funding available across public sector organisations in the UK, which will lead to questions regarding the sustainability of existing online services and digital resources.

The report is based on a study conducted by Ithaka S+R, with funding from the Jisc-led Strategic Content Alliance, which reported on findings of earlier studies showing that both funders and project leaders rely heavily on their host institutions to support and sustain digital content, beyond the end of the grant. But what will happen when the host institutions have significantly reduced levels of funding to continue to maintain and develop such content?

The report describes the need for an “early and honest appraisal of which projects are likely to require .. support post-launch“:

• Digital content, requiring just “maintenance”: These may not require ongoing growth, but certainly do require a clear exit plan to ensure that the content will be smoothly deposited and integrated into some other site, database, or repository. The issue of ongoing investment does not disappear; it just becomes the concern of the larger platform on which this piece of content now lives.
• Digital resources, requiring ongoing growth and investment: These require early sustainability planning, including identifying institutional or other partners and careful consideration of the full range of costs and activities needed to keep the resource vibrant.

The Sustainability Health Check Tool provides a paper-based checklist for those with responsibilities for managing digital content. The tool covers a number of areas including ongoing support; audience, usage and impact assessment together with preservation issues.

A series of video clips have been produced to accompany the launch of this report. It was particularly interesting to hear the comment from Prof David Price, Vice-Provost (Research) at UCL:

“We’re not just worried about things disappearing but about things never appearing! They are hosted all over the place, and not all the projects have a sustainable plan.”

This video clip is available on YouTube and embedded below.

View Twitter conversation from: [Topsy] | View Twitter statistics from: [TweetReach] – [Bit.ly]

Objectives

• to develop an application profile which enables open access repositories to expose metadata more consistently and which, in particular, conveys information about how the item being described in the metadata was funded
• to develop general guidelines for repositories which support the use of the application profile
• to support such technical development as is necessary to implement these recommendations and the application profile in common repository platforms
• to develop these such that they pave the way for a likely CERIF-based solution in the medium-long term.

Scope and approach

Funder policy regarding Open Access (OA) is being actively developed and the OA landscape is shifting. The emphasis in this phase of RIOXX is to do something which is adequate and able to be quickly implemented. This work will provide an application profile and guidelines which are inherently an interim solution. Broadly speaking, the approach we are taking is as follows:

Develop the simplest possible application profile, based on Dublin Core (DC).

Pretty much all repositories support DC, as another application profile of DC, OAI-DC, is a mandated minimum metadata format for the ubiquitous protocol for harvesting metadata from repositories (OAI-PMH). If all goes well, the development work needed for repository systems should be minimised.

Consider other, related guidelines

We have examined two related initiatives: the OpenAIRE guidelines (and the Driver guidelines which preceded these), and the EThOS Toolkit which developed an application profile of DC for eTheses.

Consider a CERIF-XML expression of this application profile

The interest in CERIF as the de facto standard format for exchanging this kind of information between systems is growing steadily. We are liaising with the CERIF Support Project and ensuring that a transition towards a CERIF-based approach remains viable.

Develop a modelled, expressive application profile

In later phases of RIOXX, we hope to develop the application profile more fully. This will take into account such things as:
* greater use of controlled vocabularies
* a move away from DC and towards CERIF
* greater involvement of systems other than repositories – notably Current Research Information Systems (CRIS).
* modelling of ‘access-level semantics’ – i.e. describing how, where and under what license or conditions the resource might be accessed and used

Rationale for some decisions in phase 1

Keeping things very simple

Timescales are very, very tight. From a pragmatic, technical point of view we have restricted ourselves in this phase to developing an approach which allows the repository to emit RIOXX records based on information properties already catered for in the repository system (that is, the placeholders for Sponsor and ProjectID already being there, even if the actual data has not yet been entered). We have deferred a more complete and complex approach to a later phase because the capacity to deliver this kind of information from institutional systems is developing rapidly.

The ProjectID property

We found ourselves unable to simply adopt the OpenAIRE guidelines as these mandate a particular syntax for the ProjectID (designed for EC funded projects) which would preclude certain UK funders. In any case, we consider it to be a mistake to embed semantics into this property and believe it is best provided as a globally-unique, opaque identifier. To this end, we are actively looking at the possibility of funders minting DOIs for the ProjectID. In the meantime, we will be requiring that the ProjectID be whatever identifier is provided by the funder of the output being described in the record.
We have chosen the term ProjectID rather than, for example, GrantID, as we have been advised that the former is the more widely used term in common usage in the UK.

The Sponsor property

For phase 1 we are mandating this property, but specifying only that a recognised form of identifier for the funder/sponsor be used. This will mean a free-text string for now. We are actively exploring possibilities for identifying and then mandating a particular authority list of funder names, such that this property becomes underpinned by a controlled vocabulary. However, this will not make it into phase 1.
This property, while essential in the short term, might become more of a convenience than a necessity, as the ProjectID becomes more reliably ‘actionable’. In the medium-term, we would anticipate being able to reliably derive the sponsor/funder from the ProjectID. For this reason, we have not modelled the relationship between these two properties closely – except insofar as they exist in a particular record. This means that some records may contain more than one Sponsor and more than one ProjectID with no direct way to relate a given ProjectID to a given Sponsor. While it would be possible to model this relationship, we have chosen not to do so in this phase, because:

• it is not the common case that a record would have more than one Sponsor
• it is more likely that a record might have more than one ProjectID, but only one Sponsor. This happens where a project has multiple versions – such as when the PI moves institution during the project.
• it is unlikely that current repository systems will be able to provide more richly modelled relationships between these properties without further development
• it is the common case that a record will have one Sponsor and one ProjectID.

We anticipate that this will need to be modelled more thoroughly in future phases.

Deferring the ‘access-level-semantics’ question

In order to convey the precise nature of the open-access ‘state’ of resource, RIOXX will need to develop a richer way of describing such concepts as ‘green’ or ‘gold’ open access, embargoes, licenses etc. The use-cases and operations which will depend on such information are not yet clear and, while the time has now come to model these, this should not be done in a hurry.

The following is a table of proposed elements and recommended formats. We propose to use extend the Dublin Core elements with two new elements under the rioxterms namespace.

• M: Mandated
• R: Recommended
• O: Optional

I would appreciate any comments people might have about the technical aspects of this.

I was asked by Ben Showers of the JISC to write a ‘challenging and provocative vision’ for library management systems, for a joint JISC / SCONUL workshop. I was given a free hand with this – the only parameters were that the piece should be non more than a side of A4 paper in length, and that it should use 2020 as its target year for prediction. I think I ignored both of these restrictions, but I had fun and it did provoke some discussion….

Dramatis personae:

• Alby, a young student & researcher in full time employment
• Charlotte, a venerable librarian
• Bob, Dan and Eva, semi-autonomous software agents

Following the unprecedented Conservative ‘walk-over’ election victory of 2015 and the subsequent consolidation in 2019, the landscape of higher-education in the UK is all but unrecognisable. The free market dominates the buying and selling of courses, and the provisioning of learning and research resources has, in the end, simply had to follow suit. Copyright has mostly been ‘fixed’ in the virtual world through a combination of an adjustment to more modest expectations of compensation for copyright holders, workable systems to control distribution, and global agreements allowing extradition and prosecution.

The student researcher (1)

Alby works, full time, as a software engineer. As part of his job, he is given some time to pursue research topics of interest to him and to his employer. His firm gives him a small budget to support this. In the evenings he studies part-time for the new Masters++ qualification. He is enrolled at three universities, visiting one of the these – the local George Osbourne University (GOU) – every Thursday evening. He finances all of this himself.

On Monday evening, when Alby gets home, he goes straight to his laptop and works through all the notes he has dictated into his smart phone during the day. He has become interested in the evolution of library systems and wants to register this interest on the Research Interest Grid (RIG). While recording notes into his phone, he has also published some of these into StreamingConscious, the latest social network to become popular with researchers, and has gained a few new connections from people with aligned interests, including a promising one with a subject librarian at GOU.

Alby then invokes his Foraging Agent, ‘Bob’. A license for Bob was given to him by a publisher, Coyote, which specialises in resources for software engineers, in return for sending him a steady stream of advertisements. Alby adopted Bob because he liked its interface, but he suspects it has in-built biases towards certain, commercial information sources. He believes that he compensates for this by carefully defining his research questions in Research Question Format (RQF) and filtering the results.

Bob runs constantly on Alby’s ‘slice’ – a portion of Personal Cloud (PC) infrastructure provided by a well-known supermarket chain. After a series of questions and answers, Bob is armed with three carefully RQF research questions, and a set of parameters, such as when to report back, and how much of Alby’s research budget to spend on a single transaction before asking him for approval. Bob has learned through observation how Alby likes to work. It knows him in a sense, enough to represent his interests when dealing with other agents. Alby then instructs Bob to begin searching, negotiating and shopping for answers, leads and recommendations, while he gets on with some reading. Alby has grown to trust Bob.

The Librarian

Charlotte is a subject librarian with many years’ experience (she tried to retire 3 year ago but has been forced to come back to work), specialising in software & systems engineering, and currently working for George Osbourne University. On Tuesday morning she checks the reports from her Listening Agents over breakfast. She controls several agents running on the library’s slice of the GOU cloud.

Bob, an agent representing someone called Alby has made contact, coincidentally, with two of her agents – one which represents GOU and which reports to her, and her own personal agent, Eva. Only yesterday, BirdSong (a social network monitoring agent) had suggested that she connect with @alby on StreamingConscious based on their mutual interest in the history of LMS systems. Charlotte’s interest in LMS systems is partly fuelled by nostalgia – she has been working with such systems for more than thirty years.

She sees that Dan, the GOU agent, has supplied Bob with material to which Alby is automatically entitled, and has automatically reserved two books from the local GOU collection for him. In so doing, Dan recommends to Charlotte the purchase of a newer edition of one of these textbooks.

Dan has also made a number of offers to Bob of more restricted material which can be supplied at a cost, including 3 inter-library-loans. Bob has accepted one of these paid-for items on Alby’s behalf and Charlotte is happy to see that it has also observed the protocol of explaining why it has not accepted the others. In one case, she sees that Bob was successful in bidding on eBay for a second-hand copy of a book which Dan had offered as an ILL. Bob has also made an offer to Dan for ownership of the book, once Alby has finished with it, in return for one free ILL. Dan needs Charlotte to approve this. However, she declines, knowing the book to be flawed, despite its 4 star popularity rating. Dan registers this decision, quietly blacklists the book against any future recommendation, and reports this decision to Bob.

Dan notes that Bob has also registered a second book on Alby’s personal virtual book-shelf and indicated a willingness to make this available to the GOU circulation agent for loan to other GOU students as part of the ‘Support Your Library’ protocol, in return for one free ILL token. Charlotte accepts this offer.

Charlotte instructs Dan to negotiate with Bob to arrange a meeting over coffee for Alby and herself. She does this partly because Eva has separately registered Alby’s interest on the RIG and it seems worthwhile meeting with Alby in person to discuss his research. She decides to investigate a couple of other suggestions thrown up by Dan in the meantime. She also notes that Dan has suggested a couple of other contacts to Bob – other people who are enrolled at GOU and whom Alby may wish to befriend on StreamingConscious – – as part of a strategy to reinforce the local GOU social network of students and researchers.

The student researcher (2)

Later on Tuesday morning, Alby wakes to find an interesting report from Bob waiting for him. He discovers he is the proud owner of a new book on LMS system design and is pleased to note that it has a four star rating – one star above the threshold he has set in Bob’s book-buying decision parameters.

Bob has, inevitably, also turned up a few offers of information and resources from the ‘invisible market’. He knows that if you have the right connections, you can get just about any book in ePub5 format. The penalties for possession of an illegally obtained, copyright resource are stiff however. Although it is not illegal, he is also a little wary of using Turpin, the global federation of Open Access papers and other resources, as he has been culturally conditioned to be suspicious of things which appear to be ‘free’.

He also finds a tentative appointment in his diary for coffee with @charlotte, the subject librarian with whom he connected yesterday on StreamingConscious. As he works close by the university, he accepts the appointment. He can pick up his reservations while he’s there.

Face to face, later that morning

Alby finally puts his pen down, and takes a swig of his coffee. He has been writing furiously for half an hour. Charlotte has just taken him on a whirlwind tour of the evolution of the LMS.

She has described how the library has learned, over the last decade, that client relationship management (CRM) is crucial to its mission. Adjusting to the new realities of social networking and global search, the LMS has become a distributed and loosely-coupled collection of processes, all designed to help connect people with resources and with each other.

Alby learns how the rapid introduction of semi-autonomous software agents into research practice took many by surprise. Although the concepts were not new, and much of the technology existed in one form or another, it took the confluence of a number of factors to finally introduce agent-mediated research:

• the cultural acceptance of an ‘always online’ culture brought about through the ubiquity of smart phones, the prevalence of global social networks and move from the desktop to cloud-based processes
• the utter complexity of negotiating through ‘permission stacks’ to determine whether or not an individual has the rights to access a given resource in a given context
• the complexity of relationships between individuals and institutions

Charlotte explains how, from having been a destination for local researchers, the LMS has dissolved into the fabric of a vast, distributed network of research interests, library collections, national, private and open resources.

While the curation of local collections remains important, the facilitation of networking, and the handling of transactions, both social and financial, has taken over as the focus of the LMS. She points out that where once it was quite easy to point to the LMS – at least as a line in a budget sheet – it has become somewhat nebulous on recent years. The LMS has become the coffee-shop of cyberspace, where software agents meet to compare notes, register interests, make deals….

Taking a sip of her peppermint tea, Charlotte sighs as she remembers how simple it all once was.

In recent years, JISC has invested in some development programmes based around an approach labelled Rapid Innovation. Rapid Innovation, in this context, described an approach of investment in small, short, cheap development projects designed to ‘scratch an itch’. There was more than an echo of the Agile Manifesto in this approach. The Rapid Innovation projects tended to show the following characteristics:

• they brought developers more to the fore
• they produced lighter, more frequent documentation
• they produced working code very early in the process
• they involved end-users directly, and throughout the project

The early work of DevCSI has been informed by this work – notably in the increased awareness of adoption of agile development methodologies.

So why is this important?

The radical changes currently being introduced to the economic and political landscape around higher education in the UK are forcing universities and colleges to re-examine themselves as ‘businesses’. With the growing interest in commodified hardware and software and remote software as a service (SaaS) options for service delivery, HEIs need to examine how they can best exploit these opportunities. (The JISC’s Flexible Service Delivery Programme has been established to help institutions in this). While HEIs will have differing levels and types of interest in what are being referred to as cloud services, they are generally going to be searching for efficiency-based savings.

The value proposition of financial cost-reduction from using shared services is something which cannot be ignored by HEIs – but it seems to me that there are some things which need to be born in mind:

• the biggest saving in cloud-based services is to the supplier, not the customer (although the supplier will pass on some of this saving)
• this whole approach is not yet well understood – especially how SaaS sits with an ‘enterprise’ service oriented architecture (SOA) approach which is also of interest to some HEIs
• some services can be outsourced more easily, or to greater benefit, than others

In The role of the central IT Services organisation in a Web 2.0 world, Joe Nicholls and David I Harrison introduce the useful characterisation of services being either chore or core. Making use of SaaS is a form of outsourcing, and outsourcing is a tricky thing to get right. There are arguments for outsourcing those things you have to do but have no special interest in (e.g. HEIs frequently outsource their catering operation). In the ICT service context such services might include the various administration systems which all HEIs need to operate (e.g. finance). These we might call chore services. However, another reason for outsourcing is a lack of capacity or expertise to deliver a service internally – whether or not that is the preferred option. Services which are core to the HEI’s business might fall into this category occasionally – even if this is not ideal. In a recession, with drastically reduced funding, HEIs might see more core services become unsustainable – or indeed need to reconsider what is core in the first place. Normally, business decisions of this sort are not so simply binary, and some complex judgement will need to be made.

Inevitably, the growing opportunity for outsourcing ICT services will be appealing to many HEIs – whether those services are outsourced to generic or specialist commercial suppliers, or to HE-sector-based consortia such as the Kuali Foundation. But outsourcing can introduce hidden costs. A lessening of control is one obvious concern. But a more insidious risk introduced by an enthusiastic embracing of outsourcing services is a temptation to start to regard the maintenance of local development expertise as a luxury. After all, if we’re going to outsource our ICT, why do we need to retain technical staff and, especially, developers. ICT is just a  commodity, right?

Well, no. I think it is a mistake to lose sight of the advantages that come from a local capacity to perform and deal with technical innovation. A local or ‘in house’ development capacity is a valuable resource in the normal run of things. In a recession, it is vital.

The successful organisation will use a recession to examine its business and to change in order to be ready to fully exploit the economic recovery, when it comes. And large organisations are getting better at preparing themselves to be able to innovate internally or locally. Scott Anthony, who has worked with Clayton Christenson who coined the expression “disruptive innovation”, lists some principles which inform an organisation’s ability to engage in innovation:

• Put the customer, and their important, unsatisfied job-to-be-done at the center of the innovation equation
• Embrace the power of simplicity, convenience, and affordability
• Create organizational space for disruptive growth businesses
• Consider innovation levers beyond features and functions
• Become world class at testing, iterating and adjusting

(I’m not entirely enamoured of the ‘disruptive innovation’ label – as my colleague Brian Kelly pointed out at the recent CETIS Conference, the HEI sector is receiving plenty of ‘disruption’ right now from political forces – certainly enough to encourage innovation!)

In Whither Innovation, Adam Cooper of CETIS asks: “Could we leave innovation to the commercial sector and buy it in?”. Answering his own question, he quotes Cohen and Levinthal (1990) who introduce the term absorptive capacity, describing :

…a model of firm investment in research and development (R&D), in which R&D contributes to a firm’s absorptive capacity….

I see a direct parallel between outsourcing too much, and losing the absorptive capacity necessary to respond to change and to innovate to meet new challenges. In my talk to the JISC Innovation Group, I presented this diagram:

This diagram tries to express the role of the local developer to act as an agent enabling and supporting change in an HEI. The developer deals with the remote, outsourced ICT system at a technical level, becoming one route through which the HEI ensures it gets the best possible value out of this arrangement. Remote services are, nowadays, guaranteed to offer some sort of application programming interface (API) which allows the more technically capable customer to tailor the service to their needs, rather than simply being obliged to use an undifferentiated, default user-interface for example. Local developers are increasingly networked with their peers in other HEIs (not least because of the efforts of the DevCSI project), so they become quite powerful in being able to exploit commonly used remote services through the free sharing of knowledge, technique and even code. And because local developers are, in some case, embracing a more agile approach to development, they become the conduit through which the end-user expresses their needs to make the remote, shared service better fit their local, idiosyncratic needs. Developers can become surprisingly aware of ‘business’ processes and information flows through an HEI, as they have to deal with them at several levels (I wrote about experiences of this sort in a previous post, SOA and reusable knowledge).

I see an opportunity for the DevCSI project to focus its efforts on this aspect of change within our HEIs. Change management is going to be crucial for HEIs as they redefine what is core and what is chore, as they decide what they can do best, and what can be best done for them by others. They are going to need a capable, knowledgeable and above all agile capacity to innovate to meet new business challenges and a changed ICT environment.

I’ve taken to using the label responsive innovation to describe the act of dealing with or instigating technical change in a manner which advances the core mission of the institution. Developers are not the only part of the solution, but they are a vital part. Not only do HEIs need to hang on to their best developers, they need to invest in them, if they are to manage change and not be managed by the changes being imposed on them.

Developers are core.

After careful consideration, I abandoned XForms and threw my energy into putting Rails on top of Fedora Repositories, which I reported on one year later at Open Repositories 2008 in a presentation called How we Integrated Fedora into Ruby on Rails and How You Can Use It.  This in turn lead to the development of ActiveFedora and the resulting Hydra framework.

I was not alone in my decision to abandon XForms.  In January 2008, the W3C published the first draft of HTML5. Later, the funding for the Mozilla XForms project evaporated.  Since then, to my knowledge, none of the browser manufacturers have commited any resources to supporting XForms.  To contrast, all of the major browser manufacturers have committed to support HTML5 (with HTML5 Forms) and have already made substantial headway in that direction.

Meanwhile, my old presentation from 2007 has been floating around the web.  I periodically get emails from people who are enthusiastic about using XForms with their Fedora repositories.  Every time that happens, I do my best to discourage them and promise to put up a blog post enumerating my criticisms of XForms. Thus, with sincere apologies for the lengthy delay, here is the quick rundown of my thoughts on XForms and a bit of info about how MediaShelf is dealing with complex, hierarchical XML in the repository-driven applications that we build.

XForms are to Beta Cassetes as HTML5 Forms are to VHS

XHTML2 and, by extension, XForms have lost their bid to become the chosen replacements for HTML4. HTML5, CSS3 and HTML5’s Forms (formerly WebForms) are the official successors, and they are looming large on the horizon. All of the major browsers already support them.   The further you sink your legs into XForms, the harder it will be to keep up when this wave hits the mainstream.

In case that’s not ’nuff said, let’s look a little more deeply.

XForms is a dead technology.

Technologies are like languages — Latin is a dead language but is still used (and useful) in rare cases. The same is true for XForms.  If you choose to use XForms, you’re choosing to use a technology that will never enjoy broad adoption and will increasingly fall short of contemporary development practices.

Losing the battle to become a standard doesn’t always kill a technology, but in this case it did. XForms was dead in the water in 2008 when the W3C decided to favor HTML5 over XHTML2. The writing was on the wall when the major corporate support for XForms quickly evaporated (ie. IBM cancelled its funding for the Mozilla XForms Project, which to my knowledge has never seen any work since).

While some organizations do still use XForms, this doesn’t mean that the technology is strong — after all, plenty of hefty corporations still run and spend money on maintaining cobol apps.

XForms makes it difficult to stay abreast of contemporary interface and interaction design.

The structure and conceptual framework of XForms feels strange and overly complex to most developers (especially developers who don’t like declarative languages). This discourages innovation on the part of those developers, who are only going to work with XForms when they are forced to. Since XForms effectively died in 2008, none of the innovations since 2007 have happened in the XForms arena. Case in point: XForms is incompatible with the vast majority of the javascript widgets and plugins floating around out there, which in turn means that unless you re-implement all of those plugins you’re stuck with kludgy, outdated interactivity in your forms.

XForms has some real design flaws

XForms assumes that In-Browser interactivity means Asynchronous XML.

With its rigid fixation on XML, XForms prevents you from taking advantage of non-XML communications.

The very notion of AJAX has evolved substantially since the inception of XForms. In fact, the very name AJAX has fallen from favor because what originally manifested as “Asynchronous Javascript and XML” has now become much more open ended. Notably, convention has drifted heavily in the direction of preferring JSON over XML by virtue of its simplicity and attendant ease of serialization/deserialization. Meanwhile XForms instead focuses on embracing the full complexity of XML, which adds complexity without much, or any, real payoff.  Further, with the arrival of HTML5, the predominant conventions, which are not all XML-centric, will become ensconced as the standard.

XForms encourages you to break the MVC pattern in your applications.

Though XForms uses an MVC model internally, it encourages you to break the MVC model in your application by forcing you to embed decisions about your XML schema(s) — which are an implementation detail of your model — into the XForms.  Thus, you’re forced to put representations of your model into the view layer of your application.  This is especially noticeable when your user wants to edit metadata that is actually spread across multiple XML documents (and schemas) in your model — though you can technically handle this in XForms with XInclude, etc, it is far from ideal.

While this is merely bad design when you’ve got just one form in one application, it compounds upon itself as you add new forms because decisions about the schema become replicated across numerous XForm controls within each of your edit views.  Any time you want to change something about your model (ie. what attribute to use on a particular metadata field), you have to update all of the corresponding references in all of your forms.

You could add a layer of abstraction in order to at least partially achieve this separation of concerns, but in that case, once you’ve added an abstraction layer, why the heck would you use XForms instead of a regular dynamic form?

How, then, to deal with complex XML?

This brings us back to the real issue at hand.  Most of the people who ask me about XForms are building interfaces for creating and/or editing complex XML metadata.  Most often they are either dealing with MODS or Qualified Dublin Core.  Since the metadata is already in XML, XForms seem like a perfect fit.

Though I was able to offer criticisms, until recently I haven’t been able to offer a complete, viable alternative to XForms.  ActiveFedora has supported Dublin Core since late 2008, but that was neither a complete nor a freestanding solution, and it wasn’t any help at all for those poor souls who are coping with MODS.  Now, as of last month, this has all changed.

As part MediaShelf’s work on Hydra for the Stanford Libraries, we have created a Ruby gem called OM (Opinionated Metadata). OM is a server-side solution for mapping between your application’s metadata vocabulary and a specific XML structure.  In order to use it, all you need to do is provide a single file specifying how your application terms map to nodes in an xml document.  Based on that information, it uses XPath and some of the characteristics of XUpdate to give you simple tools for predictably creating, retrieving, updating and deleting nodes within complex XML documents.

Though it’s only in its first stages of development, I think OM has some real promise.  We are currently using it in Hydrangea (the Hydra demo application) to edit MODS descriptive metadata and to edit rights metadata based on the Hydra rightsMetadata schema.  Also, ActiveFedora natively supports using OM to index complex, hierarchical XML into solr, so a single OM mapping file gets you all of the basics you need in order to use complex XML metadata in all of your application’s Search/View/Edit functionality while putting regular HTML (either HTML4 or HTML5) into your views.

There is much more information to come on OM.  In the meantime, go directly to the code.

The OM code: http://github.com/mediashelf/om

The Hydrangea code, which uses OM: http://github.com/projecthydra/hydrangea

Within Hydrangea, check out the OM mapping files for MODS Articles and Hydra Rights Metadata.  (Note: Both files also specify custom methods that augment the behaviors provided by OM.)  You can see how the resulting mappings (ie. [:person, :first_name] ) are used in the edit form templates for hydrangea articles.

Through the JISC-funded DevCSI project, UKOLN has been asked to arrange a ‘Developer Challenge’ for the Fifth International Conference on Open Repositories, (OR10) to be held in Madrid in July of this year.

This will be the third consecutive year that the Developer Challenge has been a feature of this conference. Previous challenges have been both competitive and creative.

Photo by Graham Triggs

This year we have been considering doing something slightly different. Previously, a general challenge has been issued, inviting developers to submit prototypes for anything which they feel is relevant and useful to the repository community. But now that the community has a better appreciation of the sort of creativity which developers can bring to these events, we wonder if we might try something a little different.

A general challenge?

We have been thinking about the possibility of the repository community issuing a particular challenge to the developers planning to attend OR10. This could be decided on by the community well in advance of the conference. If we managed to ‘crowd source’ a few ideas, we could organise a simple vote. Something we are trying to do more with the DevCSI project is to get developers together with non-developers from the same ‘domain’ (repositories in this case) – so we are quite interested in pursuing this approach with OR10.

The OR10 organisers have helpfully couched the conference itself in terms of some challenges:

In a world of increasingly dispersed and modularized digital services and content, it remains a grand challenge for the future to cross the borders between diverse poles:

• the web and the repository
• knowledge and technology
• wild and curated content
• linked and isolated data
• disciplinary and institutional systems
• scholars and service providers
• ad-hoc and long-term access
• ubiquitous and personalized environments
• the cloud and the desktop.

Perhaps one or more of these could serve as the inspiration for a more concrete developers challenge?

What this boils down to is finding a challenge in the general area of repositories, recognised as important by the community generally, which could only be met by getting developers to work with non-developers at the conference. For it to be fair, the challenge would need to be non-specific with regard to any particular repository software.

I would welcome some feedback:

• is this general approach a good idea?
• do you have any ideas for a challenge?

please feel free to comment her if you have any ideas, or alternatively drop me an email at p.walk@ukoln.ac.uk

Thanks!

OpenRepositories 2010 Presentation Proposal (Long Version)

Any repository solution provides facilities for Creation, Management, & Editing of Content as well as facilities for Searching & Browsing through that content. Experience has shown that when a solution binds these two areas of functionality together too tightly, the system becomes brittle and unworkable, discouraging innovation. Our work on the Hydra project has produced a flexible and intuitive solution that combines these two areas in an almost entirely decoupled fashion. This solution, which is already working in multiple Hydra applications, is built on a three-part pattern where Blacklight handles Search & Discovery, ActiveFedora handles Creation, Management and Editing of Content, and a small application called Shelver supplies the crossover point by indexing the content into Solr so that it will show up in Blacklight. This three-part approach reflects a strong pattern for designing and/or improving repository solutions. The main pivot of this approach is to treat indexing as its own separate part of the application and to allow that indexing processes to evolve constantly as part of the application development cycle.

This work is the product of combining established best practices, best of breed software, and lessons learned from an iterative approach to application development. While our implementation is focused on Fedora Repositories, the software could be used in multiple contexts and the pattern is certainly applicable to any content-oriented application.

The anatomy of a Hydra Application

Note: This is a working model of the functional structure of a Hydra application. The complete designs for the final features and functionality of Hydra applications reach far beyond what is presented here. For more information on the greater vision around the Hydra project, please refer to the Hydra Project pages on the Fedora Commons wiki.

• The portion of a Hydra application that handles Creation, Management, & Editing of content is provided by the Hydra Core, which consists of ActiveFedora along with a few Hydra “helpers” which integrate ActiveFedora into Ruby on Rails.
• The Search & Discovery portion of a Hydra application is a Blacklight installation - nothing more, nothing less. As with any Blacklight installation, its behavior and appearance are likely to be customized but otherwise there is nothing Hydra-specific about it.
• Shelver (which can be run either from within the application, from the command line, or as a JMS listener) indexes content and its salient metadata in Solr, usually pulling that content from Fedora.

These three components — Blacklight, and Hydra Core and Shelver — work in concert to present a consolidated repository solution to the end user. Meanwhile, the three components are sufficiently decoupled that each could be run as a freestanding application. They interoperate based on a minimal contract that revolves around decisions about what information should be in Solr and how it should be represented in the Solr index in order to achieve the ideal search experience.

In the process of customizing or extending a Hydra application, some changes require modifications to all three components, but most changes impact only one or two of the components at a time. This makes it very easy to iteratively improve the application and adapt to real world needs.

This structure grew naturally out of a process of exploration. In early 2009 developers at UVA and Stanford discovered that it was relatively easy to put Blacklight on top of a Solr index that had in turn been been populated by ActiveFedora — effectively turning Blacklight into a search & discovery interface for that Fedora repository. Based on that, we tried dropping ActiveFedora-driven views & controllers for editing Fedora content into the same Ruby on Rails application as an existing copy of Blacklight. It worked like a charm. The two systems happily coexist. What we found was that as long as we could change and refine how the metadata percolates from Fedora into Solr, getting Blacklight to operate together with the ActiveFedora management component was completely straightforward.

With most Hydra applications, all content is stored in a Fedora Repository. However, there is nothing to prevent you from adding non-Fedora content to solr and having it show up in the (Blacklight) search & discovery views. Of course, that content will not be editable unless you implement the code to integrate with that content’s host system.

Best of Breed: Blacklight & ActiveFedora

Blacklight is a next generation Search & Discovery tool. It was intentionally designed to serve a single purpose - Search & Discovery - without having any knowledge of indexing, cataloging, or even the location of the content it’s searching through. Whatever information you have in your Solr index, Blacklight will help you expose a rich, faceted search interface for exploring through that information and displaying detail views of individual records. This open-ended design made it very easy for us to integrate Blacklight directly into our Hydra applications as-is. The ease with which we achieved this seamless integration is a testament to the quality of Blacklight’s design.

ActiveFedora is a Ruby library that encapsulates the details of interacting with a Fedora repository and provides high-level tools for defining data models, creating Fedora objects, and modifying the data associated with those objects. While opening the door to rapid, iterative application development, ActiveFedora also attempts to expose and accentuate many of the strong design patterns inherent in Fedora. ActiveFedora’s emphasis on flexibility and design patterns provided us with many opportunities to make our Hydra applications robust and re-usable. In particular, ActiveFedora makes it possible for multiple Hydra (and non-Hydra) applications to operate on top of a single Fedora repository, thus achieving the goal of providing many lightweight views onto complex, heterogeneous repository content.

Hydra Core: the building blocks of an interface for creating & editing Fedora content

Hydra Core provides the Ruby on Rails code that handles Creation, Management, & Editing of Fedora content. This primarily consists of Rails helpers for generating edit interfaces and Rails controllers to handle the submissions from those interfaces.

Fedora allows a great amount of freedom with respect to data models and metadata. As a result, we could not simply create a single generic content management interface in Hydra. Instead, we created a number of “helpers” that allow you to deal with your Fedora content and its metadata at a high level of abstraction. For example, the editable_metadata_field helper generates the HTML for displaying an editable version of whichever metadata field you specify. All you have to know is what field you want to display and where it is stored within the object. Everything else is handled for you.

The forms generated by the Hydra helpers need somewhere to submit their data to. This is handled by the Rails controllers provided by Hydra core.

Underneath the helpers and controllers, Hydra Core relies on ActiveFedora to handle connecting with Fedora, modeling Fedora objects & metadata, and performing the basic operations of creation, retrieval, updating and deletion.

Shelver: a script that brought unexpected freedom

When we wrote Shelver, we didn’t anticipate how integral it would become to the application development process. Shelver started out as something extremely simple. A developer at Stanford initially wrote it in order to populate a Solr index with some working data from a Fedora Repository. Over time, as needs arose, we built out the script to be more robust. It soon became apparent how crucial it is to be able to modify and/or augment the behavior of your indexing tool. In most other systems, the indexing tool is either implicit (relational databases) or external to your application and difficult to re-configure (ie. Fedora GSearch). As a result, when working with other systems, discussion of (and changes to) the indexing strategy are kept to a minimum. In contrast, since we had Shelver at our disposal, we found ourselves constantly tweaking it to satisfy new functionality. This ability to tweak our indexing routines gives us radical freedom to explore new features, improve the search experience, and increase the quality of search results.

Eventually, we pulled shelver into the Hydra app itself so that we could trigger it as part of the save/update process, though we retained the hooks for running it as a command line tool as well. We also did this because we found that changes we made to shelver often corresponded to changes in the search interface. Shelver was continually evolving in conjunction with the application, so it made sense to track the code with a single versioning system.

Approaches to Indexing: from RDBMS to Fedora + Shelver

RDBMS (data model & search index combined)

If you rely solely on a Relational Database to drive your application, your data model (the database schema) is also your indexing model — any search oriented changes necessitate changes to your data model. This makes it difficult to refine and extend the search & discovery portion of the application without impacting other areas of functionality.

RDBMS + Solr (separate search index from data without much thought to the conceptual differences)

A number of tools exist for pulling content from a relational database into Solr. This achieves the goal of separating the search index from the data itself, allowing you to have an indexing model separate from your data model. However, often with these systems the indexing methodology remains tightly bound to the data model. This is more of a conceptual stumbling block than a technical one. It’s easy to underestimate the complexity and distinctiveness of indexing for search & discovery. It is not enough to index your data in Solr; you must think differently about how and why you put it there. This “thinking” must be manifest somewhere in the application’s code, ideally separated from the rest of the application.

Fedora + GSearch + Solr (freestanding tool specifically handles indexing)

Fedora is explicitly designed with the idea that you should separate your data model from your indexing solution(s). This allows us to use any variety of content models and metadata schemas to represent our content in Fedora while pulling that information into any number of indexes to suit specific searching needs. The most common indexing approach with Fedora repositories is to use Fedora GSearch to pull Fedora content into a Lucene, Solr or Zebra index. This approach has the benefits of completely separating the data from the index while also providing a freestanding, configurable tool to handle the process of indexing.

GSearch was designed with the goal of enabling 1) full-text searching of Fedora content and 2) indexing of arbitrary XML metadata from Fedora objects. It runs as a web application alongside Fedora, listening for JMS messages or REST API commands telling it to (re)index Fedora objects. The process by which GSearch indexes the content is implemented as a mix of XSLT and Java code.

GSearch establishes the strong best practice of decoupling both the search index and the indexing process from the data itself. This pattern was part of Fedora’s design all along, but thus far GSearch has been the clearest manifestation of it.

Because it was designed specifically to enable full-text indexing using XSL Transformations (XSLT), GSearch operates on the premise that you are transforming the content in order to put it in the index. In a basic system, transformations are sufficient. However, most repository solutions eventually need to actively process the data when indexing it, performing complex actions in order to decide how to populate the search index. Because XSLT does not lend itself to performing such complex processes, you must modify Java code if you want to implement this type of processing in GSearch. Modifying that code has proven daunting for most. Very few projects have taken on the challenge of modifying the GSearch code itself. Those that have modified the code have only done so in minimal and relatively stable ways.

Fedora + Shelver + Solr (allowing the indexing methodology to constantly evolve)

If you want to provide a great search & discovery experience in your application, you must make it easy to iteratively “massage” the indexes. Anyone who manages a Blacklight or VuFind installation on top of their ILS (or anyone who participates in Code4Lib) can attest to the fact that in order to achieve a truly successful search & discovery experience you must continually refine the way you index your metadata. Little changes in your indexing methodology can bear tangible results for end-users.

In building SALT, the first Hydra application to combine Blacklight with ActiveFedora, we created Shelver as an alternative to GSearch because we wanted to be able to specify our indexing process in Ruby code and, where possible, we wanted to use simple mapping files rather than being forced to use XSLT and Java to perform those actions. We assumed that Shelver would be a relatively simple application whose code rarely changed. After all, when using GSearch we rarely changed the XSLT and basically never changed the Java code. We expected that the same would be true with Shelver. We were wrong. Shelver is constantly changing because we are constantly coming up with new things that we want to do to improve the search & discovery utilities in our Hydra applications. As time passes, the code of Shelver itself has stabilized, but the instructions for how to index specific data from Fedora continually morphs as a regular part of the application development process. In fact, touching the Shelver code has become such an integral part of our work that we can’t imagine building a repository solution without this kind of freedom.

Conclusion, Observations and Best Practices

To review, some of the recommendations coming out of this work are to

• use indexing as the crossover point between decoupled solutions for searching through and managing your content
• make the indexer an explicit, evolving part of your application
• use flexible components that were designed with iterative development in mind
• re-use established best practices where possible
• combine best of breed solutions for astounding results

We were pleasantly surprised to discover how easy it was to combine Blacklight and ActiveFedora into a single Fedora solution. The three-part pattern that emerged out of this effort, which now constitutes a basic Hydra application, builds on well established practices and serendipitously combines them in a stable, intuitive way. This in turn provides a strong base for us all to carry out a great amount of innovative work in the coming years.

Linked Data was the dominant theme today, starting with the pre-conference Linked Data Meetup which included presentations and workshops from Tom Scott, Patrick Sinclair, and Silver Oliver of the BBC and John Sheridan & Jeni Tennison of data.gov.uk among many, many others.

Lin Clark of DERI ran a Drupal RDFa workshop in the afternoon. She demonstrated RDF Mapping, the RDF SPARQL Proxy, and the RDF SPARQL Endpoint modules and gave a preview of the RDF features to expect in Drupal 7 core.

One session I really wanted, but wasn’t able, to attend was Jim Downing’s Multicore Processing session. Not a terribly interesting title, but he worked through mining linked data from data.gov.uk, using the Clojure library Incanter to find correlations in the data, and finally, parallelizing the data processing.

The dev8D spirit of developer happiness was nowhere more evident than in this year’s new attention to electronics. I sat at a table watching Emma Tonkin build an electronic whiteboard with a nano-projector and Wiimote. Walking over to the projects room, I saw Ben O’Steen working with facial recognition software, Dave Tarrant turning off and dimming lights over the local network, Julian Cheal wiring each room’s entry with an RFID reader, and Ian Ibbotson programming an Arduino. Tomorrow, I’m told, we’ll have someone showing off a RepRap 3D printer.

I find the impromptu meetings as valuable as any of the structured components of dev8D. I sat down with Andy Turner, a researcher at the University of Leeds, who introduced me to GENeSIS, a mad-scientist level modeling and simulation project that I thought was absolutely fascinating. Just as serendipitously, as we wrapped up Day 1 of dev8D at the pub (where else?), I found myself drinking a pint with Scott Wilson and we got to talking about Apache Wookie and HTML 5.

See you tomorrow!

If you have not yet taken a look at CDL's curation micro-service specifications, most of which may be printed on as few as one or two sheets of paper, see the Digital Library Building Blocks.

My co-workers in the LC Repository Development Center have been chatting about these specs on and off throughout the year. After months of procrastinating, I finally read all of the specs on Thursday; it's wonderful that you can do so in the course of one reading session, I might add. Yesterday a bunch of us RDCers got together to chat (informally) about the specs: what they're for, how they work, and how they interact with one another. I learn by doing, by examples, so I combed through each of the specs in advance of our meeting and tried to construct a minimal repository[1] based on micro-services.

Here is a tree visualization of the final product, inevitable warts and all: The services I used were Namaste, Content Access Node (CAN), Pairtree, Dflat, Reverse Directory Deltas (ReDD), Class-based System for Managing Object Properties (CLOP), and BagIt (co-developed by LC and CDL).

As I mentioned in our Friday meeting, recounting my experience exploring the specs: the bad thing is that I spent an hour building a repository with rudimentary tools such as mkdir, touch, cp, ln, and emacs; but the good thing is that I built a repository in one hour using common, rudimentary tools. It's a very compelling paradigm. Ed's already built a tool implementing some of Dflat, further demonstrating how lightweight these micro-services are. (UPDATE: Ed notes that this code is a work in progress and is "barely functional.") (UPDATE 2: The dflat library has come a long way. Check it out if you're interested. Also, I just committed a pretty basic Namaste library: http://github.com/mjgiarlo/namaste. Only took about an hour, which is a testament to the power of lightweight specs.)

I am certain this will be a running thread at work as the specifications evolve and our understanding of them grows. Some questions and comments that occurred to me while exploring the micro-service specs and building the minimal repo:

• CAN was a bit puzzling. The spec is simple enough, but I found some of the conventions confusing, and I was left wondering what CAN provides other than a container. What I would like to see is a simple use case and perhaps more examples. Thus, the CAN stuff in my sample repo doesn't feel very useful only because I had a hard time working with the spec.
• CLOP feels like the least mature of the specifications. It seems generally useful to be able to put digital objects, however you define that, into classes and define properties on those classes. The spec did not clearly convey to me just how it accomplishes that aim. A few examples would go a very long way. I've got some CLOP stuff in the sample repo but I have no idea how close my implementation matches the spec.
• Is Dflat dependent on ReDD? One would assume not since there's an optional property in the dflat-info.txt file for specifying a delta scheme. But, say, could you stub out the v001 directory (reserved to hold the initial version of a digital object) and use a system such as git or bazaar?
  
  One might argue that these established delta schemes, if you want to call them that, have many more developers and users than a system such as ReDD and thus should persist longer and have more tools built around them. I imagine the micro-service viewpoint would acknowledge that point, but counter that the spirit of these specs is to avoid dependencies from outside the filesystem?
• Is the ReDD specification meaningful outside of a Dflat given that any one ReDD directory knows nothing of its successors and predecessors, or is it dependent upon Dflat?
• Could a BagIt bag live inside of the ReDD reserved "full" directory? That is, could the "full" directory be marked up appropriately to be a BagIt bag?
• How many tools exist for these specs? I notice there's code in CPAN for Pairtree and Namaste, which is a fabulous start. Tools are the difference between YAMF (Yet Another Messy Filesystem) and reliably managed curation services. Granted, tools such as cp and emacs already exist and are part of the appeal of these micro-services, but there's also tremendous room for error if operations are all done "by hand."
• To what extent has CDL transitioned to using these specs/tools?
• Are other institutions using these specs/tools? I have heard tell that digital library folks from the University of Michigan and the University of North Texas may be involved.

I hope I don't sound overly critical. I'm really glad our colleagues at the California Digital Library have written these specifications and applied their deep experience to what could be a transformative paradigm[2] in the digital curation world. Kudos to them!

Notes

1. Perhaps it's more in line with the specs to refer to this space as "a managed filesystem that drives repository and curation services," given the CDL philosophy that preservation is not a place/repository. But it's easier to say "repository," so there you go.
2. Please excuse the fanboyishness; this filesystem fetishism is exciting stuff!

The survey closed in July, and the subgroup spent August writing a report on the survey results. That report is now final and it's available to the public. Feedback may be sent to our (woefully underutilized) public i2info mailing list, left as a comment on this post, or e-mailed to me privately which I can forward to our internal list.

The next step is to build upon the report to draw yet more conclusions from the data — there's an awful lot there — and flesh out some repository use cases for institutional identifiers. The I2 core group is moving quickly towards finalizing identifier metadata elements so that a standard may be drafted, and I think having some use cases documented will help drive the standard in a direction the community can get behind.

Onward and upward.