This blog post is about the functionality I have added to DSpace to allow it to accept deposits from within Microsoft Word using SWORD.

If you are unaware of the authoring add-in, then before reading the rest of this blog, take a look at Pablo’s YouTube video ‘Integrating with repositories and journal submissions’ at http://www.youtube.com/watch?v=2_M2gfUyVzU. The video explains the authoring add-in, so I’ll not duplicate that information in this blog post. The rest of this post explains how I extended DSpace to work with the add-in…

In order for DSpace to be able to ingest a package, it needs an ingester that understands the format and knows how to unpack it and extract the metadata and file(s). In the case of .docx files created by Microsoft Word, it needs to know how to extract the metadata from within the file, and to archive the file as-is. This is a pretty easy task as a .docx file is actually just a zip file (try renaming it from .docx to .zip and then take a peek inside!). So I wrote an ingester than unzips the file, extracts the NLM metadata that the add-in inserted in the file, and then creates a new DSpace item with that metadata. Finally it adds the complete .docx file as a bitstream for people to download.

Some of the metadata such as the authors identities are held in the .docx file is held in the customXml/item*.xml files, and other parts such as the article title and abstract are held in the actual document contents in word/document.xml. The ingester extracts these values for use in the new DSpace item.

<w:t>Add an S to Microsoft Word and you get SWORD</w:t>

<my:name.>
<my:name.content-type.datatypeattribute.attribute.></my:name.content-type.datatypeattribute.attribute.>
<my:name.name-style.datatypeattribute.attribute.></my:name.name-style.datatypeattribute.attribute.>
<my:surname.>Lewis</my:surname.>
<my:given-names.>Stuart</my:given-names.>
</my:name.>

I then configured the DSpace ingesters to use the docx ingester when it encountered .docx files:

plugin.named.org.dspace.content.packager.PackageIngester = \
org.dspace.content.packager.PDFPackager = Adobe PDF, PDF, \
org.dspace.content.packager.DSpaceMETSIngester = METS, \
org.dspace.content.packager.DSpaceDocxIngester = DOCX

I then configured the SWORD package to expose the fact that it supported .docx files in its SWORD service document:

sword.accept-packaging.Docx.identifier = application/vnd.openxmlformats-officedocument.wordprocessingml.document
sword.accept-packaging.Docx.q = 1.0

Finally the DSpace SWORD interface needed to know which packager to use for .docx files based on their MIME type:

plugin.named.org.dspace.sword.SWORDIngester = \
org.dspace.sword.SWORDMETSIngester = http://purl.org/net/sword-types/METSDSpaceSIP \
org.dspace.sword.SimpleFileIngester = SimpleFileIngester \
org.dspace.sword.DocxIngester = application/vnd.openxmlformats-officedocument.wordprocessingml.document

All that is needed to use this is a copy of the authoring add-in (http://research.microsoft.com/en-us/projects/authoring/), and a suitable formatted template for the repository that you wish to deposit the document into (dspace-swordapp-org.docx). The template is preconfigured to deposit directly into the DSpace SWORD demo repository which I have upgraded with the new code to accept .docx deposits. Feel free to create an account in that repository, install the add-in, load the template, and try out a deposit!

This complete end to end process allows you to create Word templates, and to mark them up with required and optional fields. It also allows you to embed details of the SWORD deposit repository URL (so the users do not need to know what it is) within the template for easy deposit. This could be used for example for a journal editor to provide a template and a deposit location for new paper submissions all-in-one. And this use case could be extended: for example if a faculty member wants all their students to submit an assignment with a template, they could do so and use the repository as the end point rather than a traditional VLE. And unlike a VLE, the repository will probably provide search and indexing facilities across the deposited documents. I’m sure as this tool gets used more, there will be a lot of new ideas for how it can be used.

Comments welcome!

Workshops on Search Engine Optimisation (SEO)

As described on the JISC Digitisation blog the Strategic Content Alliance (SCA) are running a series of free workshops entitled “Improve your online presence“. The workshop series, which will be held in June and July in London, Edinburgh, Cardiff and Belfast, is being coordinated by Netskills. The workshops will “introduce simple and inexpensive search engine optimisation techniques to improve your online presence, web visibility and website traffic“. I will be contributing to the workshop content by running a session on the role of the Social Web in enhancing access to scholarly and cultural content.

The Potential of Twitter

The potential of Twitter was recently discussed in a post entitled How much is it worth to be one of Twitter’s suggested users? which was published in the Guardian’s Technology blog. As described in this post, being included in Twitter’s Suggested Users List can boost one’s numbers of followers, and thus traffic to links included in the tweets being published.

Coincidentally on Friday 5^th June 2009, whilst accessing this blog’s administrators interface in order to delete one or two spam comments which had failed to be detected by the Akisimet spam filter, I noticed that the top three referrers for the day were from the Twitter Web site (from Twitter.com, twitter.com/home and twitter.com/Twitter_tops). On further investigation I discovered that a page on the Twitter Web site which provides links to resources about use of Twitter had included the following link to a post on this blog:

What is Twitter? It’s An Interactive Business Card: http://cli.gs/YL6R4D –Share this article: http://bit.ly/180g9w

Now although this link resulted in driving the most traffic to the blog in over 3 weeks, this was disappointing to me. I had been after evidence that Twitter can provide successful in driving traffic to arbitrary resources, rather than just traffic to an article about the Twitter service.

However a better example was provided by the blog statistics for UKOLN’s Cultural Heritage blog. As illustrated the statistics for May 2009 showed that, after Google, the second most popular Web site for driving traffic to the blog was Twitter.

In this particular example the most popular post in the month was one on Explaining the Risks and Opportunities Framework- a blog post which was announced on Twitter at 08.55 on 21^st May:

Blog post explaining the Risks & Opportunities Framework published at http://tinyurl.com/p72kld

Evidence, it would seem, that Twitter can enhance the visibility of one’s Web content and therefore provide an example I can use in the workshop. But what of the dangers of using Twitter in this way? Might not Twitter followers resent being used as fodder for marking materials? Isn’t there a danger of killing the goose that lays the golden eggs?

Twitter Optimisation

Although some people regard Twitter as being essentially an informal communications channel and a tool for community building we can now observe that it is being used for a much wider variety of purposes. But what are the emerging best practices which one should adopt in order to optimise Twitter’s potential to maximise access to ’stuff’ out there, as opposed to engaging with one’s Twitter community?

Keep it short: Perhaps the best advice is to keep your tweets short to allow other to retweet (RT) the message, perhaps including their own comments.

Acknowledge the limitations: If you do intend to use Twitter as a one -way publishing mechanism (as, for example, the MLA does) then you need to recognise that you should not expect to gain the benefits which fans of Twitter, as described in a post entitled “The person is the point” by Mike Ellis, feel they gain from its use as an individual.

Consider publishing a policy: You may also wish to consider having a policy covering your use of Twitter, as described in a recent post on “Emerging Best Practices For Institutional Use of Twitter“.

Think about your followers: If you are using Twitter as an individual but also wish to promote areas of your work you will need to consider the balance between engagement (chatting with your mates), support (helping your mates), requests (asking your mates for held) and dissemination (telling your mates what you’ve being doing and what you’re proud of). This was an area I addressed in a post on “Twitter Can Pimp Up Your Stuff – But Should It?“.

And if you’re still sceptical that Twitter has any significant role in delivering traffic to a Web site I’d suggest you read the TechCrunch article “For TechCrunch, Twitter = Traffic (A Statistical Breakdown)“.

A report entitled “The Edgeless University: why Higher Education Must Embrace Technology” was launched earlier today. As described on the JISC Web site:

The Edgeless University argues that technology in higher education is not just about virtual learning environments, but is increasingly central to the way institutions provide learning and facilitate research. Technology is making research and learning possible in new places, often outside of institutions. Far from undermining them, this is creating exciting opportunities for universities to demonstrate and capitalise on their value so will take strategic leadership from inside institutions, new connections with a growing world of informal learning, and a commitment to openness and collaboration. This is the radical role of The Edgeless University.

I haven’t yet had a chance to fully absorb this 90 page report but there were a number of aspects to the report which reflect my areas of interest. I should first disclose, however, that I contributed to the report (Peter Bradwell, author of this DEMOS report, was aware of my work in this area and invited me to give my views).

The need for fundamental changes in the higher educational sector: The report describes the comment made by one participant at a roundtable meeting who described the current predicament of the higher education sector: ‘This seminar feels a bit like sitting with a group of record industry executives in 1999’. The report went on to say “It is no use lamenting the golden age of universities (or record companies). The goals of the two ‘industries’ remain the same, but they must refocus on how to achieve them. Society’s aspirations for the sector remain the same. The challenge for institutions is to find the way to do it.“

The need to understand changing student expectations: The report quoted an interviewee who said “Technology is part of people’s daily life in a university, I would say everywhere except in the classroom” in order to illustrate the need for institutions to “get better at understanding exactly what it is these students need” .

New tools to support teaching: It was interesting to note that the report, in a section on how social media tools can support collaborative teaching described Michael Wesch’s work at the University of Kansas in the US in using using online tools for collaborative and team-based student coursework including tools such as sites such as Netvibes, Yahoo Pipes and Diigo. Although I’m pleased to see Web 2.0 tools being highlighted in the report, it was somewhat strange to see a US-based example of use of these fairly mainstream tools. Aren’t there similar examples to be found in UK HEIs?

“A renewed commitment to openness“: The report includes a section with this title. The opening quotation for the section “Science is as much about conversations in corridors as it is about papers in journals” strikes me as summarising the benefits which the Social Web can provide for the research community. However this section seems to focus more on the ease of access provided by tools such as Scribd and iTunesU rather than the issues if open acces and open data.

“Experimentation and investment“: I was p[particularly pleased to see that JISC Developer Happy Days’ (Dev8D) being mentioned as an "event brought together communities of coders and users from educational software and beyond" with the aim of "mix[ing] people interested in civic society with those who have the skills to develop tools to encourage social change“. Dave Flanders (now of JISC) will be pleased to see that his work in bringing together a set of developers has been appreciated in this report.

A few weeks ago the “Higher Education in a Web 2.0 World” report was published. And today we see another report which provides a similar top-down view on the importance of Web 2.0 in higher education. If you encounter resistance to change from senior managers in your institution I’d suggest you beat them over the head with these two report until they realise that Web 2.0 is changing the higher educational environment.

Near the end of my strawman post, I wrote:

The I2 repositories subgroup will be sending out its survey on identifier use cases in the coming week. It will be interesting to see if the requirements we have thus far identified still obtain in light of the data we collect from the survey.

We completed the survey late last week and began distributing it. Here's what we sent out:

The NISO I2 Working Group is surveying repository managers to determine the current practices and needs of the repository community regarding institutional identifiers. We value your time and your input in the process to create a standard for a new institutional identifier. We hope that you will complete the survey which should take less than 15 minutes. The survey will remain open through Monday, July 6th.

Here is a link to the survey: http://www.surveymonkey.com/s.aspx?sm=RGQgZ3090DVrb3kFzr3P3Q_3d_3d

Please feel free to share this message with other interested parties.

First we used Survey Monkey to send the survey link to approximately one-hundred repository managers that the subgroup identified. Our process for identifying repository managers involved pulling together a list of prominent repositories from subgroup members, and then gathering more from OpenDOAR, "an authoritative directory of academic open access repositories." Then subgroup members were encouraged to share the survey link with colleagues, and post it far and wide via blogs, listservs, and tweets. The listservs we targeted were: JISC-REPOSITORIES, metadataLibrarians, digital-curation, SPARC-IR, ir-net, REPOMAN-L, PALINET-IR-L, dspace-general, fedora-commons-users, DC-IDENTIFIERS, and code4lib.

I've already received a few responses and have gotten useful feedback. Two of the hardest questions to answer so far have been: "What is an institutional identifier?" and "What is a repository?"

Institutional identifier

An institutional identifier is defined as a symbol or code that uniquely identifies an institution. Domain-specific examples of existing identifiers include SAN, IPEDS, GLN, MARC Org Code, and ISIL. Another example might be a Handle prefix or ARK name authority assigning number.

Repository

Institutional repositories and subject repositories like arxiv.org are clearly 'repositories', but beyond that it is a somewhat ill-defined term. One might look to the Kahn-Wilensky architecture, or the OAIS reference model (PDF), or even Wikipedia for definitions, but it's not clear that even the authorities agree on what constitutes a repository.

It's a system. It's network-accessible and typically has a web interface of some sort. Files and groups of files sometimes known as objects tend to be deposited in them, perhaps for some combination of management, access, or preservation. Many run Fedora, DSpace, and ePrints, and factor heavily in scholarly communication. Some are document-centric. Some will accept anything. To some, a learning management system may be a repo. To others, a content management system may fit.

My background is in academia so my own definition is somewhat based in that context, but I wouldn't say the term is necessarily limited to that context. There are other NISO I2 scenarios for library workflows and electronic resources, so it's safe to assume that repository does not mean ILS or OPAC or ERP system. My hope is that folks have their own working definitions of the term and can decide for themselves what it means.

We've given folks a little over two weeks to respond to the survey, so the constant I2 drum-beating will quiet down for a while around here. I am very interested in what sorts of responses we get from the survey. Fun times!

Oh, and perhaps it goes without saying, but if you're a repository owner, manager, expert, developer, or stakeholder with an interest in identifiers, please feel free to take the survey!

A Hero For Our Sector

One of the real strengths of the UK higher education sector is the way in which we can work together as a sector, meaning that the whole is geater than the sum of the individual parts. This is undoubtedly true of JISC (which is envied in the higher education and research sectors around the world) but also applies elsewhere. One example of this is Hero: “the official gateway to universities, colleges and research organisations in the UK“: a gateway funded by the various funding bodies (HEFCE, SHEFC, HEFCW and DENI) and supported by other higher educational agencies and by the high educational institutions themselves (and note that I was involved in the technical advisory group for the “HE Mall” as it was originally called.

Indeed will a service such as Hero, why would higher educational institutions wish to use other channels for online marketing, particularly social networking service such as Facebook which, despite its popularity are, in some circles, regarded with suspicion in not hostility?

Our Hero Is Dead …

Alas for those who believe that the sector should own its marketing channels, the Hero.ac.uk service was closed on 4^th June 2009 (and the image shown above was taken from Hero’s most recent entry in the Internet Archive, from 10^th February 2008) I should disclose that last year I was interviewed by a consultant who had been appointed in order to identify future directions for the service, including whether the service was viable. I pointed out the flaws in the Hero service: it did not have the community aspects which potential new students might expect and it was a ‘walled garden’ – information could be uploaded to the service but there were no easy ways of getting the data out again. “Make ‘Hero 2.0′ a trusted service which could host structured institutional data“, I suggested “and provide APIs to allow developers elsewhere to add value to the service“. But this did not happen.

… Long Live a New Hero?

If the managed service to promote UK higher educational institutions is too costly to provide, why don’t we appropriate popular social networking services to fulfil this role? This is an idea inspired by a Tony Hirst’s post on “Appropriating technology” which he described as “appropriating technologies that might have been designed for other purposes in order to use them in an educational context” but I would replace ‘educational context‘ by ‘marketing context‘.

And, if we’re honest, isn’t Facebook the new Hero? It can provide the popular service for hosting institutional marketing materials. And it can provide the community aspects which Hero failed to provide. Admittedly it may be a ‘walled garden’ – but then so was Hero, so nothing is being lost.

But if we wish to use Facebook in this way, don’t we as a sector need to identify the best practices for making use of Facebook, including minimising the risks associated with the service? And shouldn’t we be exploring the benefits which might be gained by working collaboratively?

Some initial thoughts on this:

Institutional URL: As mentioned in my recent post on “Have You Claimed Your Personal And Institutional Facebook Vanity URL?” we are seeing Facebook URLs being minted as a single string (edgehilluniversity) and words separated by dots (aberystwyth.university). We might wish to consider whether there are advantages in seeking agreement on the form of the name – perhaps even using an institutional domain name in the URL (e.g. www.facebook.com/www.bath.ac.uk). However it is probably too late to do anything about this (which arguably demonstrates the failure in having not had such discussions previously).

Trademark disputes: We’ll want to avoid the possibility of trademark disputes. Might we see one between Leeds Metropolitan University and say, Loyola Marymount University over www.facebook.com/lmu?

Ownership of Facebook resource: Who has access to the institutional Facebook account in your institution? And what if they’ve left or you can’t find the owner? The information should be regarded as a valuable institutional resource and ownership should be managed appropriately.

Workflow processes: There’s a need to establish effective workflow processes for information provide on the institutional Facebook page. Ideally information would be hosted elsewhere and automatically updated in Facebook though use of, for example, an RSS application in your Facebook page.

Will Facebook pages enhance or diminish Google Juice: Might not institutional content which is replicated on Facebook pages diminish institutional ‘Google juice’ as my colleague Paul Walk has suggested? Or, alternatively, might content held in popular services such as Facebook and Wikipedia (and previously, to a lesser extent, Hero) held to increase traffic to the institutional Web site? Indeed if such replication of content is felt to be counter-productive, shouldn’t institutions try to prevent Web sites having links to their content rather than seeking to maximise such links?

Facebook Terms and Conditions: It would be useful to gain a better understanding of the Faceboom terms and conditions and the implications for an organisation’s pages in order to inform appropriate risk management approach. If the concern is that Facebook will claim ownership of marking material provides, is that really of concern?

Explore Possibilities for Facebook Applications: Might there be benefits in developing Facebook applications to make the UK HEI pages more appealling?

But have we, in the UK, missed the boat? Looking at the timetable for the forthcoming Eduweb 2009 conference I notice sessions on topics such as “Facebook — a case study of building virtual relationships“, “Cheap, Fast, & Out of Control: Brand management & recruitment..” and “Recruiting and Marketing in the Web 2.0 World“. We’ve nothing along these lines planned for IWMW 2009 – but as the bar camp sessions can be submitted at the workshop itself, perhaps there’s an opportunity to build on these ideas?

Oh, and if you think it is inappro[oriate for an organisation to make use of a social network in this way, look at what companies such as Starbucks and McDonalds are doing on Facebook.

[Series]

In the prior I2 post, I wrote about the requirements the repositories subgroup has come up with for an institutional identifier standard (with the hope that our findings re: repositories could be generalized to other scenarios).

My strawman proposal of sorts is to explore how well linked data patterns fit this problem space. Linked data, briefly, is a way to expose and link data on the web in a more semantically meaningful way, and is often summarized using the four principles put forward by Tim Berners-Lee:

1. Use URIs as names for things
2. Use HTTP URIs so that people can look up those names.
3. When someone looks up a URI, provide useful information.
4. Include links to other URIs. so that they can discover more things.

That's the crux of it. Linked data takes well-known patterns on the web (linking, dereferencing, etc.) and applies them to data, which in this case could be metadata for identifying institutions.

Let's examine each of the requirements and the applicability of linked data thereto.

1. Should be agnostic to type of institution, e.g., libraries, museums, personal collections, historical societies: The web is already agnostic to type of institution. HTTP URIs do not favor one type of institution over another.
   
2. Should handle varying institutional granularity, e.g., institution-level, campus-level, division-level, unit-level: HTTP URIs are flexible in this regard. Hierarchy, should one wish it to be surfaced in the identifier, may be encoded in either a DNS hostname or the path appended to the DNS name. One can imagine a URI like "http://department.division.institution.tld/unit/subunit" or "http://institution.tld/campus/office/individual".
   
   Hierarchy needn't be surfaced in the identifier if one favors opacity, in which case "http://registry.tld/xnjsdasd" would suffice as an identifier, and may instead be entirely reflected in the (RDF) representation returned by dereferencing the URI.
   
3. Should handle linking among institutions and subordinate units: Linked data handles linking via well-known HTTP mechanisms, referenced in the fourth principle of linked data. Unlike the HTTP link, which has limited semantics, linked data links are semantically rich and extensible.
   
4. Should express different sorts of relationships among these institutions and units: The "useful information" in the third principle of linked data is typically provided by an RDF representation, which is itself a list of assertions. These assertions, or triples, consist of subjects, predicates, and objects. The ability to express the relationships in this requirement is limited only by the availability of vocabularies that contain sets of predicates and classes for subjects and objects. Think of the predicates as elements defined within a metadata standard, e.g., Dublin Core "creator", MODS "relatedItem", and so forth. Vocabularies that contain these predicates and classes are growing and evolving daily, and should there not be a vocabulary that contains the relationship one wishes to express, it is fairly easy to create a custom vocabulary.
   
   The ability to mix and match vocabularies provides an expressiveness that is often not found in document-based metadata formats and the flexibility to express radically different relationships on a per-industry or per-institution basis. This latter point is important as the I2 group has identified both core metadata elements for identifying institutions of different types and additional elements for specific types of institutions. Why re-invent a new metadata format or schema when all one needs to express may already be contained in others?
   
5. Should relate to existing relevant identifiers and registries: Same as requirement#4. Linked data is all about expressing relationships between things, e.g., institutions, identifiers, registries, etc.
   
6. Should be globally unique: HTTP URIs are guaranteed to be globally unique by virtue of the distributed DNS system and hierarchical naming within each HTTP service.
   
7. Should be actionable: HTTP URIs provide dereferenceability/actionability via the well-known HTTP protocol.
   
8. Should enable retrieval of metadata sufficient to identify the institution, which may vary widely by institution: HTTP URIs are actionable per requirement #7 and the metadata returned is flexible per requirement #4.
   
9. Should accommodate changes as institutions come and go and re-organize and be able to relate defunct institutions to new ones: Linked data patterns provide for redirecting from defunct representations (institutional identifiers) to new ones via HTTP redirects. One may also add assertions to institutional metadata such as owl:sameAs, for instance, which says that the institution identified by the given URI is the same as another institution identified by another URI.
   

This seems like a compelling path to follow for the I2 standard.

The I2 repositories subgroup will be sending out its survey on identifier use cases in the coming week. It will be interesting to see if the requirements we have thus far identified still obtain in light of the data we collect from the survey. If so, I would like to explore the idea of linked data for institutional identifiers a bit more.

The JISC is funding a landscape study on the UK HE sector use of content, communication and social networking services developed by commercial companies (or, perhaps more accurately, outside of the JISC sector).

As we know although JISC has developed a number of services specifically for use within the UK higher and further education sector (e.g. Jorum, JISCmail, etc.) people within the sector are increasingly using services developed outside the sector, either in addition to – or in some cases instead of – JISC-provided services.

Since evidence of this usage is fragmented and often anecdotal, the JISC SIS Landscape study aims to provide a snapshot of the current situation in the UK.

My colleagues Ann Chapman and Rosemary Russell are leading this work and have set up the JISC SIS Landscape Study blog to facilitate their work. We welcome contributions to this blog in order to collate evidence on how such services are being used within the sector. Please note that JISC are primarily interested in use of such services within the UK higher and further education sectors. If you are outside this sector, feel free to contribute but please make it clear in your comments the sector you work or study in.

The I2 IR scenario subgroup approached the issue of institutional identifiers in repositories by first brainstorming about the various issues, problems, and sticking points that make identifiers in this space (and elsewhere) such a complex topic. Folks on the subgroup are repository managers or are otherwise involved with or knowledgeable about the repository space, so the brainstorming exercise yielded a good number of concerns.

The purpose of the exercise was to enumerate concerns and issues that could inform a draft survey to be administered to repository managers and experts around the globe in different organizational contexts: libraries, subject disciplines, archives, historical societies, etc. The purpose of the survey is to get an idea of the use cases and constraints around institutional identifiers in these different repository contexts, the assumption being that we ought to have requirements grounded in real world usage before we go off building a standard.

I will note here that the subgroup has worked up a draft survey that has just recently been reviewed by a small group of folks who know about survey design, and we hope to administer the survey to the aforementioned Reporati this week[1]. Which is to say that I don't yet have a strong grasp of the use cases out there in the wild, and this series should be construed as my own premature cognitive fumblings. But let's assume for now that what we learn from the survey results matches our initial brainstorming exercise.

Here is a slightly modified and boiled down version of the concerns and issues the subgroup came up with for a potential institutional identifier standard, which resembles a set of minimum requirements:

1. Should be agnostic to type of institution, e.g., libraries, museums, personal collections, historical societies
2. Should handle varying institutional granularity, e.g., institution-level, campus-level, division-level, unit-level
3. Should handle linking among institutions and subordinate units
4. Should express different sorts of relationships among these institutions and units
5. Should relate to existing relevant identifiers and registries
6. Should be globally unique
7. Should be actionable
8. Should enable retrieval of metadata sufficient to identify the institution, which may vary widely by institution
9. Should accommodate changes as institutions come and go and re-organize and be able to relate defunct institutions to new ones

I doubt the list is exhaustive; I am almost certain we will uncover all sorts of tangly and esoteric use cases that add requirements. I expect it. Why else would we be gathering to discuss the need for an institutional identifier if it were a solved problem or a simple one? [2]

Nevertheless, looking at the above list, the task we've taken on starts to feel less onerous. And thinking about identifier systems constrained by the list of concerns, the mind starts to cook up all sorts of possible solutions. I'll share one in the next post in this series, a strawman proposal of sorts, and how it addresses each of these requirements.

Notes

1. We will also x-post to repo-related mailing lists as well, and some of us may blog or tweet about it. My inclination is to cast as wide a net as possible so as not to miss important use cases. We can always scope things out later on, but it's useful to be inclusive at this point lest our own assumptions carry the group forward.
2. The cynical among you might have interesting answers to this question.

In the comments, Geoff Hutchison makes an excellent point : -

I think these are also key points:

• Software releases also rarely garner wide recognition, and without an accompanying paper to cite, the work cannot be paid back in citation currency.
• Additionally, emphasizing one’s work on software or publishing methods papers carries a risk of being perceived as a programmer first and scientist second, falling on the wrong side of the technician/scientist divide.

I have been told to be sure to publish “real chemistry” more, even though I want to get “citation currency” for open projects. After all, without publications or citations, I can’t get grant support for these activities either.

So is the trick to get grants for “real research” that have “broader impacts” in software development?

I sympathize with Geoff. I have friends and colleagues whose academic careers have been made difficult, jeopardized or even ended because they gained technology expertise in order to make their research more effective, and ended up being perceived as too techie to be “proper scientists”.

The problem, as I see it, is the very existence of a “scientist/technician” divide. I’m beginning to suspect that the problems software engineers have with existing metric systems based on journal articles are more the symptom than the cause of a lack of respect and recognition afforded to technical staff in academia, and that fixing it through software journals and the like will be a temporary palliative cure.

Perhaps I’m just getting more bolshy the more streaks of grey appear, but perhaps academia needs to change to embrace software engineering, rather than software engineering twisting itself out of shape to fit in with the anachronisms of scientific research.

Admittedly, I’m pretty short on “how”. The publication / citation model clearly isn’t working so well for researchers any more - perhaps fundamental changes there will help. Funding of the sort JISC, and the e-Science programme certainly help provide support and some credibility, but you still need a Principal Investigator, and if they’re academic they risk being dragged toward the divide for their pains. Post-grad training could also help to change the culture from the ground up. The JISC-CRIG Dev8D initiative is a band-aid for this very problem; an extra-mural support network to give developers some well-deserved recognition they don’t necessarily receive in their own institutions. Beyond vague ideas and short term fixes, I’ve got nothing.

It’s an important issue though, because so much of modern research depends on software development, and when (not if) the software / technical talent walks out of the door to work in industrial research (where it is respected), how will academia compete?

For the first half of the BOF we sat in mixed groups, eating our lunches and each talking about the work we do. We then split up by language (Ruby, Python, PHP) and discussed language-specific topics. For that second half I sat at the Ruby table where we talked about ActiveFedora, JRuby, RDF support for Ruby, MODS support for Ruby, Solr (solr-ruby and RSolr), and how Blacklight fits into the mix. 

I closed the conversation by asking if we should set up email lists for collaboration. It seemed reasonable to set up a general mailing list for the solutions community as well as a list specifically for people doing stuff with Ruby, Fedora repositories, and (most likely) ActiveFedora. I also resolved to encourage the creation of Python-oriented and PHP-oriented equivalents. For now I have created two lists on Google Groups. The first one, Fedora Commons Create, is for general discourse about creating client applications for Fedora. The second, ActiveFedora / Ruby + Fedora Commons, is for Ruby-specific collaboration.

In the end, I was really pleased to realize that for the first time we had a substantial group of people interested in each of the main interpreted languages (Ruby, Python, PHP) and each group had at least one open source Fedora-based project to use as a starting point for their conversations. The Ruby group had ActiveFedora, the Python group had Ben O’Steen’s work and Peter Herndon’s Django integration, and the PHP/Drupal people had Islandora & Fez to start from. 

This was a comfortable step forward from the scenario as it was a year ago.

Fedora Commons Create

Visit this group

ActiveFedora / Ruby + Fedora Commons

Visit this group

So @cardcc suggested at #rpmeet that most everyone was using an offline template spreadsheet to mark #jiscbids and that it would be a real time saver if there was a way to auto copy and paste the offline spreadsheet in one go. Well I don’t have a one go solution but I do have a three step solution that I was able to piece together over the weekend (while marking bids).

DISCLAIMER: THIS IS ONLY A PROTOTYPE AND IS NOT THE OFFICIAL WAY TO SUBMIT MARKS FOR JISCBIDS, THIS IS ONLY AN PROTOTYPE TO SHOW HOW THIS MIGHT BE ACHIEVED. YOU STILL MUST SUBMIT BIDS USING THE CURRENT APPROVED FORM.

None the less, if you would like to help me tweak this form so that it could be useable in the future then please follow the below instructions:

1.) Download this Excel Spreadsheet. This spreadsheet is the marking template. Use the template spreadsheet exactly as it is set up. If you change any of the headings rows or columns it will render the spreadsheet useless.

2.) Once you have completed filling out the template spreadsheet follow this screencast (w/audio) step by step for uploading to GoogleDocs. The above screencast will also show you how to publish an RSS feed from the document. TO NOTE: this RSS feed must be a “cell” rss feed so the data can be sliced and diced in multiple way so that it is exposed on the backend system to multiple outputs. Eventually, I’ll enable it for just the simple url for the SS, for now please just make sure you are exposing the right RSS feed.

3.) Once you have the RSS feed go to this form and fill it out. Job done, well then there is a lot of back end processing that takes place to parse your spreadsheet into the various forms it needs to go to as part of the next step in the marking workflow.

Resources used in this work:

• http://docs.google.com/support/bin/answer.py?hl=en&answer=75507y
• jQuery and the joy of arrays

Podcast notes: Mark Shuttleworth, Canonical, free software and wealth creation

Originally uploaded by dff.jisc

itc.conversationsnetwork.org/shows/detail3994.html

* The stimulus of innovation is disclosure: not only saying what but
how we created the innovation
* how can we drive innovation faster?
** architect platform to be extensible and imbeddable, e.g. mozilla addons
** platform tolerance, must embrace our enemies (windows) to embrace
all humans, be benevolentt
** bring in fresh blood, it is not neccessarily the core that makes
things happen
* methodology and software development processes
** purpose of methodology is to harness talent
** aggregate development by building interest
** agile for community (not pair prog) by phil oconnor
** architect by community and collaboration
* permission free development to enable web services
** tools must interoperate over the web, eg bugzilla, trac, launchpad
** branching should be enabled to any commmunity
* change in innovation drives economics
** economic models: advertisement, hardware providers pay for
innovation… not likely
** pay for service provision?
* can a community create the cutting edge experience… better than Apple?
** "we should be building software that helps it users get laid"
* conical investing in a beautiful experience.

Podcast notes: IT conversations, Anne Thomas Manes, ‘Is SOA Dead’

Originally uploaded by dff.jisc

itc.conversationsnetwork.org/shows/detail4004.html

* We have been doing "Service Oriented Integration" which has been
focused on integration rather than architecture
* Intel paper on IT business valuation for metrics that effect bottom
line: how many people, how many hours, etc
* biggest impediment to SOA is cultural and political, not technological
** the question for those not obsessed with technology is: what is
going to increase bottm line?
* business managers and leaders won’t understand SOA, developers are
beginning to understand and so are building it from the bottom up.
SOA itself will very seldom ever happen as a top down political
process.
* we *as humans* suck at architecture.
* enterprise architecture is completly different to application architecture.
** enterprise architecture is about reducing cost.

The following projects have been accepted:

• Andrius Blazinskas Fedora DAO implementation for DSpace, beta release
• Gaurav Kejriwal Collection Administration Enhancements
• Ashly Markose Report Generation Tool for DSpace
• Bojan Suzic DSpace REST webapp

Congratulations to Andrius, Gaurav, Ashly and Bojan!

Features

• SWORD support has been upgraded to the new 1.3 version of SWORD.
• Sitemaps.org sitemaps, and robots.txt files have been added to the XML (Manakin) user interface.
• DOI links can now be rendered in the JSP user interface.
• The uketd_dc OAI-PMH metadata format is now included as standard. This is required by UK institutions who want to expose their electronic theses to the EThOS service. No additional code is now required.
• A new statistics collection sub-system has been written to help collect statistics in a better fashion.
• A better PDF extraction utility (xpdf) can now be used to extract the fulltext out of some files where the current system (pdfbox) fails, or uses too much memory.
• The exporter tools will now export nested files correctly (e.g. where you have archived a website). It also now exports file descriptions that you may have set.
• The exporter tool has a new option (-m or -migrate) which strips out any per-repository-specifics which would get re-created when re-ingested. This is useful if you pre-load content into a test repository, and then want to move them into a production repository without taking with them items their temporary handle, provenance information and deposit dates etc.

Security

• Prior to 1.5.2 items that were restricted (anonymous access disabled) were still accessible via OAI-PMH, RSS feeds, and daily subscription emails. There are now configuration options to disable each of these.
• Provenance information used to be included in the metadata that was exposed in the <head> element of the item web pages (for consumption by search engines and tools such as Zotero). The provenance information is not needed, and has been removed.

Authentication

• Shibboleth support has been added.
• Support for Hierarchical LDAP servers has been added (where users are spread across branches of an LDAP tree, rather than existing all in the same branch).
• Automatic groups for all LDAP and Password users have been added, allowing to create groups for all authenticated users, without having to add them to a group manually.
• IP authentication now works for users who are not logged in, as well as for users who are logged in. Prior to 1.5.2, users had to first log in, before they could access collections or items that were protected by IP address. It now works in a better fashion so that, for example, internal users can access such materials without having to have an account.

Internationalisation

• New or updated translations of the software have been added for Thai, Italian, Ukrainian and Greek.

Bug fixes

There have been many bug fixes, but of note are:

• SWORD deposits are now a lot faster. For example a deposit of a 10MB PDF file on my development server used to take about a minute, it now takes 9 seconds.
• Statistics now work correctly in the xml user interface. Prior to 1.5.2 they did work, but unfortunately looked in the wrong directory for the reports, so the reports did not show.
• If you have multiple authentication methods available (e.g. LDAP and Password Authentication) you won’t suffer any more database connection leaks.
• The handling of UTF-8 has been standardized throughout the xml user interface.
• When non-admin users deposit items using the xml user interface, the admin menu options no longer appear immediately following a deposit. Non-admin users could never use these links, but they shouldn’t have appeared.

For a neat little overview of the changes in 1.5.2, see this Wordle made using the CHANGES file from DSpace:

I am now working as a programmer in the Digital Services Team at the University of Auckland Library. The team is a multi-disciplined group of people working on all things digital within the library. As the biggest university in New Zealand, this involves many dozens of projects and lots of new and exciting things. In the five days I have been working here already, I have already started working with EAD xsl files for DigiTool, extraction of XMD metadata from PDF files, DSpace item migration tools, and research management systems. Lots of new fun stuff to learn about!

No doubt there will be many more blog posts on my work as it develops further.

Catania, Sicily, Italy

Open Grid Forum 25th Conference (OGF25)

It’s not entirely clear when I figured out that I was sitting on a standards body panel discussing the creation of a digital repository related standard. I’m pretty sure it finally clicked sometime after the session was over, once I had consumed a couple glasses of wine.

I still don’t see what I contributed to the conversation, though the other participants assured me that my comments were useful. The experience reminds me of my friend, let’s call him Josh, a community organizer who was recently pulled onto one of the Obama administration’s advisory panels. Shortly after joining the advisory panel, Josh confessed that at the end of most calls he has to follow up with a friend and ask “Ok. So what exactly did we just decide and who is responsible for doing what?”

The panel discussion started by making observations that we’re all familiar with:

• the importance, and associated challenges, of unique identifiers and persistent URIs
• search, retrieval, and management are separate concerns, each with appropriate standards associated (ie. SWORD, OpenSearch, etc.)
• cloud computing is very different from cloud storage

After this, I quickly found myself in completely unfamiliar waters when conversation abruptly turned to the creation of a standard for digital repositories. I thought “Pshaw. We don’t need Yet Another Standard. Where did this come from?” In fact, the whole field of repositories is so new that the prospect of a repository standard seems absurdly premature to me. Discussion on the panel honed in on the two obvious contenders for a standard: 1) metadata requirements, and 2) functionality profiles (a list of features necessary in order for a repository system to be deemed compliant and interoperable). From my perspective, repositories already swim in a glut of metadata standards (as well as non-standard, ad-hoc metadata) and, by nature, must embrace heterogeneous metadata. The second notion, that of functionality profiles, sounds like something that few will read and none will understand. To be honest, the entire discussion confused me. I did my best to contribute to the discussion where I could.

After the workshop ended, I had a chance to catch my breath and discuss the panel with a couple of people. Eventually, I came to look at the whole scenario from a different perspective and had a mild change of heart. In a discussion with Neil Chue Hong, a very smart guy from Edinburgh, I started thinking about all the informal conclusions that frame discussions between developers at conferences like Dev8D, Code4Lib and RepoCamp. I then thought about all the little architectural wins and failures that I see in software like Flickr, YouTube, Hulu and (woah) ABC’s full episode player. After all, these are repositories too. Within a few moments of pondering, an initial list of obvious basic guidelines shone through quite clearly.

• give permanent, unique URIs to all content you expose, even if you intend to limit access to that content based on geography or time of access
• support linking with versioning or datetime info
• expose a RESTful API
• give preference to AtomPub
• consider ORE when you need to express aggregations of data
• provide linked data (RDF) endpoints

Some open topics also seem like obvious fodder for discussion:

• what query language(s) to use in search APIs
• navigating the difference between standards and interoperability
• leveraging standards where possible (ie SWORD)

These are merely the things that seem obvious to me right away. What would happen if we got the really smart people talking in this vein?

I think this warrants further exploration and, strange though it is, I expect that the outputs of such exploration might resemble the stuff of standards bodies (be it a recommendation, a community document, or a standard). Possibly I have been infected with that odd standards-wonk bug, or possibly I’m just catching up with the rest of the world in acknowledging the inevitable.

Catania, Sicily, Italy

Open Grid Forum 25th Conference (OGF25)

[Note: I'm posting my backlog of updates from the past 2 months of travel. An update specifically about the OGF Repositories Workshop will follow shortly]

I made it to the conference center in Catania, Sicily a few hours before the OGF Repositories Workshop. Immediately upon arriving I met Nick Ferguson, coordinator of the workshop, and had a nice chat with Neil Chue Hong about repositories, ORE, and grid computing vs. cloud computing. After that I was left to kill time until the workshop by sitting in on one of the OGF sessions. At first, I stepped into what I thought was the Earth Sciences session, but it turned out to be the Computational Chemistry session and went way over my head. I then passed through a handful of other random presentations before settling on a room where about 30 people were having a discussion about XQuery.

I soon discerned that this group was hammering out the spec for some sort of standard data systems interface. When I arrived, they had been debating the strengths and demerits of XPath/XQuery vs. SQL as a query language. The converation quickly stumbled into the pit of interoperability hell. Standard interjections abounded: “Some implementations won’t have that data to return…”, “you will have to expose user info in order to support that…” mumble mumble “… we didn’t do it that way because one unnamed vendor couldn’t support it…” I nearly laughed out loud when an attendee from the back of the room interrupted the discussion declaring “But in most situations, you should only be returning items owned by the current user.” 

I still had no idea what data they were attempting to expose. (I later learned that it was the RUS-WG, who are defining a standard interface for retrieving job usage records … Obscure indeed.) The 90-minute discussion ended up having nearly nothing to do with the actual data these people want to work with. Instead, the conversation was entirely dominated by the travails of navigating the strange space of Data API design.

Meanwhile, serendipitously, I was using this downtime (and the conference wifi access) to finally read George Thomas’s slides about recovery.gov publishing open data. Though I missed the presentation, the slides spell out the project’s intentions pretty clearly. They’re full of references to REST, ATOM, RDFa and the LOD cloud. I experienced such a fascinating contrast between the exposition before my eyes and the discussion filtering in through my ears. In particular, one of Thomas’s slides jumped out at me. The slide, titled “Follow the dollar, not the person”, showed a semantic model for users, user groups, and posts in a bulletin-board style Community Forum system. It was totally readable, totally understandable, precise, flexible, and using an ontology that lends itself to re-use.

Over the past year, I have satirically placed a golden halo above “linked data” in my mind. As I sat in the RUS-WG session, light fell upon that halo and it glowed.

This experience, as well as consequent discussions at OGF, has left me with a distinct sense that there’s a pattern here. We are all, of our own accord and in our own little techno-fiefdoms, attempting to do the same things and running into the same challenges. I think that the previously obscure field of digital repositories has valuable perspective to provide and many pieces of wisdom to share in this domain. I hope to see more public discourse about these topics, and I know who to start prodding to speak up. Watch this space.

Post Script:

The morning following my OGF session-hopping experience, I realized that the track I had passed over, innocuously titled “HEP”, was a meeting of the High Energy Physics community. In particular, it was primarily a discussion about how they are going to handle processing the data outputs from the LHC experiments when they fire up the collider later this year. /me kicks himself for missing this.

Make sure your pom has a build section that points to your clojure code in a scriptSourceDirectory element (and?) or in a resource element. You can then start a Repl with your dependencies on the classpath by running

For extra credit, include jline:jline as an optional dependency in your pom and change your execution line to: -

The brain-child of David Flanders and Ben O’Steen, with support and ideas from several others and funding from the JISC, dev8D has been a fantastic success, and has managed the difficult task of appealing to, and being successful for, a range of people with varying levels of experience and technical chops. The inexperienced developer looking to be exposed to new ideas and to the wisdom of more experienced folk was well served. Julian Cheal of UKOLN fitted this description and he embraced the opportunities dev8D presented to him, engaging at all levels with the event to the extent that he was rewarded with both a prize for his helpfulness and a special mention at the awards dinner for ‘best newcomer’. But the older hands were fully engaged nonetheless – presenting on their areas of expertise in ‘lightning talks’ in the true barcamp style which geeks have embraced as their own way of conducting conference sessions. It was great to see so many familiar faces together at one event, being unashamedly techie, exchanging ideas and help.

Although, like some others, I was forced to miss some of the event due to a deadline for bids to a JISC call falling on the Wednesday, I still managed to sit in on some sessions, and I learned plenty, especially in a talk on agile development about which I’ll blog more, separately.

One of the things which stood out at dev8D was the way in which users (or UberUsers) were invited to engage with developers. There’s an important, non-obvious distinction here. Users were invited to come into the developers’ environment. Brave users, you might say! Normally, developers are invited into the users’ environment…. for just long enough to explain to them what the users require. Users would often rather not have to deal with developers all that much. To step into an environment of happy, busy developers must have been an eye-opening experience for those users who were brave enough, and open-minded enough to try it. Although I wasn’t on the ‘dragons den’ panel looking at the prototypes being developed in the Developer Decathlon, it was remarked to me several times that the quality of submissions was better than in previous events – and that this was attributed to the fact that users had been involved in the prototyping process. I’m one of the judges who’ll be marking these submissions and I’m really looking forward to seeing what was produced.

With these events, there are little things which can make a difference. The use of Wordle to produce personalised name badges for each delegate was inspired, as was the use of happiness tokens to reward help or ideas. The Twitter back-channel was used to tremendous effect – the ‘#dev8D’ tag made the top ten Twitter ‘trends’ worldwide. Sam Easterby Smith even built a Twitter-powered developer-happiness meter!

I’d like to go on record thanking David Flanders in particular for driving this event – the guy must be utterly exhausted after working 18 hour days for a week. I think we should also recognise the vision of those in the JISC (and especially Rachel Bruce) who were prepared to back what must have looked like a risky proposition. There was value in the event itself – the networking, and the capacity building which went with this and I have good reason to believe there will be value in the prototypes and ideas generated as a result. But, perhaps most importantly, the sector has just shown the world that it values its developers, and is prepared to invest in them, and even spend a little to make them happy. I believe this will have been a wise investment. As I said on Twitter, there’s a community developing which I’m proud to be associated with.

If you want to know more, the tag ‘dev8D’ has been used extensively in various systems. Some examples:

• Photos on Flickr
• Google blog search

Image by Dave Pattern (http://www.flickr.com/photos/davepattern/3274205523/sizes/m/)

Repository Road

Originally uploaded by davidflanders

Came accross this the other day, I’m sure it will show up in many
presentations from hence forward.

Dear CRIG,

We would like to cordially invite each of you to a 5* Awards Dinner on Thursday night February 12th at the stunning Court Restaurant atop the British Museum Reading Rooms.

If we could ask you to please RSVP at the below link. Also prior to RSVPing could you consider whom to nominate for the award of “exemplary innovation in the community” (the nomination is part of the RSVP form).

http://is.gd/ey8P

Also as part of the developerHappinessDays event (Feb 9-13th), there will be a final CRIG barcamp on Friday (13th) at Birkbeck college where lead developers for EPrints, DSPace, FedoraCommons and Microsoft will be present to discuss the latest advancement in repository innovation. If you would like to attend this day please sign up for dev8D below and select the “roll your own day” and put down “repository summit”:

http://www.dev8d.org/book.html

Hope to see you all there.

Kind Regards,

CRIG Support Team

From the website:

Over four intensive days we’re bringing together the cream of the crop of educational software developers along with coders from other sectors, users, and technological tinkerers in an exciting new forum.

Share your skills and knowledge with the coding community in a stimulating and fun environment and come away with new skills, fresh contacts – and you might even win a prize.

The top ideas generated at the event will be documented, publicised and made available to the community.

This innovative event includes:

• Pre-event workshops with code labs to share knowledge and develop skills
• The ‘developer decathlon’: a two-day team coding session with prizes for the best code
• A community-building day bringing together developers based around software platforms
• Roll-your-own-event: space and time for your own community-specific event or workshop

I think this will be an excellent opportunity for developers from the education sector especially, but also from beyond this, to get together and share ideas and experience. And drink beer.

It’s all happening in London, in and around the Birkbeck College, between the 9th-13th February.

The tag for this event is ‘dev8d’.

With the news that arXiv will now accept deposits from institutional repositories, Dorothea Salo continues her theme about a deposit flow which goes from author, to institutional repository, to subject/discipline repository. Dorothea offers some scenarios, including:

Achaea University adopts a Harvard-style open-access mandate. If she wants her articles in arXiv as well, Dr. Troia must rather annoyingly dual-deposit… unless Achaea’s IR implements a deposit pipeline to arXiv, in which case the most she has to do is tick a ticky-box (and I can imagine ways to abstract away the ticky-box).

In an abstract sense I appreciate the notion of the ‘deposit pipeline’. I also agree with the main point which is about the direction of the flow. Indeed, I have previously characterized the institutional repository as being, or more usually containing, the source repository. However, I remain slightly doubtful about the need for the flow to be initiated by the source. If there were some mechanism by which the subject/discipline repository could be alerted to the appearance of relevant materials in the institutional repository, then doesn’t it make sense for the subject repository to fetch the record/artefact, rather than wait to have it sent. Well, we already have the mechanism, it’s called RSS (or Atom) and it’s already supported by some of our most popular repository software.

Come to think of it, an even better approach might be for the subject repository, having been alerted to a new & relevant deposit in the institutional repository, to simply maintain a pointer to the original (optionally creating new and related resources)

In other words, as a certain generation of programmers would put it, pass by reference, not by copy.

• http://www.slideshare.net/jimdowning/web-feeds-and-repositories-presentation
• http://www.dspace.cam.ac.uk/handle/1810/206424

Earlier this month Richard Wallis of Talis interviewed JP Rangaswami at BT, and posted a podcast of the conversation. Sterling stuff - I thoroughly recommend listening to it in full. I’ve pulled out some of the bits as quotes here.

If you work in a very vendor dominated world you can abdicate responsibility for a lot of what you do by transferring not just the risk, but the worldreward to the vendor. That doesn’t scale any more.

If a problem is generic, look to the open source community to solve it. If it’s a narrow market for the problem … then look to the commercial environment to solve it. If it is unique to your enterprise, you’d better solve it yourself, because no-one else is going to solve it for you.

We’ve lived through a whole generation of mistakes when we had proprietary architectures for the way we had information in enterprises. First you paid money to completely drown the information in concrete, then you paid money to dig it out to move it somewhere else. That’s what enterprise application integration looked like, spending money to sticking it into somebody’s silo then spending even more money taking it out of silos. Instead of exposing data you were excavating data, and paying for the privilege of your own data. That is the danger we face if we don’t get issues to do with identity, with authentication and permissioning, with intellectual property rights correct in this generation. Because we will end up repeatedly wasting money digging out stuff that should have been made available much more cheaply because the costs of reproduction and transmission are going down.

Wishing I could self-replicate and get to Online Information (as well as going to the DCC Conference) to hear JP speak there!