The Linking You project was provided by the University of Lincoln and funded by the JISC under the Infrastructure for Education and Research Programme. Its aim was to look at and make recommendations for improving the way that identifiers for .ac.uk domains are planned and managed in higher education institutions. The background to this work was described on the project web site:

The web is now fundamental to the activity and idea of the university. This Toolkit provides a standard way of thinking about your institutional URI structure, making it easier for people (and their browsers) to both remember your web addresses and locate where they are in your web site. It also helps prepare your institution for the world of linked data by proposing a clear and concise model for your data, making smooth integration with other systems easier and faster. A good URI structure can be easily understood by both humans and machines.

Although one of the benefits which implementation of the report’s recommendation was to “Improve discoverability of resources (and SEO)”, the Linking You project focussed primarily on identifiers for resources hosted within an institutional domain. This post aims to complement the Linking You work by gathering evidence on additional aspects: gaining an understanding of the size of institutional web sites, measuring the numbers of links to the institutional home page and other resources hosted within the domains and variants of the URI for the most important page on a web site – the institutional home page.

About Blekko

A few weeks ago James Burke (@deburca)  introduced me to Blekko: a “search engine that slashes out spam, content farms, and malware. We do this by having a smaller crawl of 3 billion pages that focuses on quality websites. We also have a tool called a slashtag that organizes websites around specific topics and improves search results for those topics.“ In response to the question “What information is available on the SEO pages?” the site describes how:

blekko doesn’t believe in keeping secrets. As part of our effort to make search more transparent, we provide a view of the data that our crawler gathers as it crawls the web.”

Every blekko search result has data associated with it that you can see. You can access it by either clicking on the SEO link tool in the second line of each result or else searching with the /seo slashtag. For example, apple.com /seo

Further information about Blekko (although it spells its name as ‘blekko’ on its web site I’ll use ‘Blekko’ in this post) is available on Wikipedia.

Using Blekko to Analyse Russell Group University Web Sites

What might Blekko tell us about UK university web sites? Blekko’s SEO pages provide details of the following information: geographic link distribution by state and country; inbound links; duplicated content; page source; sections and site pages. Blekko was used to survey the 20 Russell Group university home pages. The survey was carried out on 2 January 2012. However on 24 January it was noticed that the host rank, numbers of site pages and numbers of inbound links had changed significantly from  702.3 to 205.4, 945 to 8,406 and 24,442 to 627 respectively. The findings were rechecked but no other significant changes were noted.

The results are given in the following table.  Note that you need to be logged in to the service in order to view the results.

Note that in the above table explanatory notes are given for figures displayed in braces e.g. {0}. Also note that the Universities of Newcastle and Nottingham both seem to have 16,041 pages.

A Tale of Two Domains

Whilst carrying out this survey it was noticed when checking inconsistencies that different results were obtained when using variants of the domain name and the institutional entry point. The following table lists known domain name variants. Note that the main domain was taken from the address given on the Russell Group web site.

It should be noted that although the table describes the institutional part of the domain which is taken from the Russell Group web site, the analysis is carried out using www.official_domain.ac.uk  In two cases the well-established www. prefix was not used. These were www3.imperial.ac.uk and www2.warwick.ac.uk. However for the analyses the www.  prefix was used as it was felt that this would be the form used by the majority of users.

Discussion

The Blekko web site contains a page which summarises its core principles, which include:

Quality vs Quantity: blekko biases towards quality sites. We do not attempt to gather all of the world’s information. We purposefully bias our index away from sites with low quality content.

Source based, not link based: blekko does NOT rely solely on link based authority.Too many people engage in efforts to game search by linking for purposes other than navigation. blekko relies on human beings and their judgement of the authority of sources to dictate search results.

Open and Transparent: blekko makes freely available to its users all of the data that provide the underpinning of our search results. This includes web data, ranking information and the curation efforts of our users.

Blekko would appear to have a role to play in providing universities  (which are unlikely to use ‘black hat’ SEO techniques such as use of link farms) with a better understanding of their visibility to search engines. However, despite the commitment to openness and transparency, the Blekko web site does not appear to provide details of their ranking algorithms.

Despite the current difficulties in interpretting the host rank in the above table, the information is provided as a snapshot, which may prove useful if Blekko subsequently do publish details.  Of perhaps greater interest is the site pages column which, it would seem, contains the number of pages which have been indexed.

There does appear to be a significant diversity in the size of the Web sites, ranging from 4,727 (for the University of Liverpool) to  507,319 (for UCL), although apart from these two outliers the size of other Russell Group university web sites ranges from  9,679 and then 9,679 to  20,053.  These figures seem to suggest that there may be differing patterns of uses for institutional Web sites, ranging form the small and managed provision of focussed resources through to a more devolved approach. However although the managed approach would appear to have benefits, it does lead to the question as to where resources and services which are felt to be useful to the individual researcher or academic or their department should be hosted, and whether policies which acts as barriers to the creation of resources on an institutional service will result in content being hosted on cloud services.

Further interpretation of these findings will probably require an understanding of the institutional web environment.  However one aspect of the survey which does not require an understanding of the local context is the numbers of links from external services to the institutional web site. Links from authoritative web sites to a web site can influence the discoverability of the resources. A more detailed survey of such links will be published shortly.

Paradata: As described in a post on Paradata for Online Surveys it is important to document the tools and methodologies used when gathering evidence based on use of online tools in order that findings can are reproducible. In addition possible limitations in the tools or the way in which the tools are used should be documented.

This survey was carried out using the Blekko web-based service over the first two weeks in January 2012 and the findings rechecked on 25 January 2012 and changed recorded. Links are provided to the  results provided by the service. However in order to view the findings you will need to be signed in to the service (the service is free to join).

The findings for the University of Birmingham had changed significantly over a period of three weeks. It is not clear whether the variation was due to changes in the University of Birmingham web site, an artefact of the multiple domain names and entry point URLs for the University of Birmingham home page (http://www.birmingham.ac.uk/, http://www.birmingham.ac.uk/index.aspx, http://www.bham.ac.uk/ and http://www.bham.ac.uk/index.aspx all resolve to the same page) or changes at Bleeko (e.g. reindexing the web site).

Note that the form of the domain name give on the Russell Group University Web site has been used. This is normally based on the full name, with the exceptions of Cambridge (which uses “cam.ac.uk”), Edinburgh (“ed.ac.uk”), Glasgow (“gla.ac.uk”) .

The results for the host rank are based on an undocumented algorithm. The information on the size of the site is dependent on the number of pages which are harvested. prefix was used as it was felt that this would be the form used by the majority of users.

Dramatis personae:

• Alby, a young student & researcher in full time employment
• Charlotte, a venerable librarian
• Bob, Dan and Eva, semi-autonomous software agents

Following the unprecedented Conservative ‘walk-over’ election victory of 2015 and the subsequent consolidation in 2019, the landscape of higher-education in the UK is all but unrecognisable. The free market dominates the buying and selling of courses, and the provisioning of learning and research resources has, in the end, simply had to follow suit. Copyright has mostly been ‘fixed’ in the virtual world through a combination of an adjustment to more modest expectations of compensation for copyright holders, workable systems to control distribution, and global agreements allowing extradition and prosecution.

The student researcher (1)

Alby works, full time, as a software engineer. As part of his job, he is given some time to pursue research topics of interest to him and to his employer. His firm gives him a small budget to support this. In the evenings he studies part-time for the new Masters++ qualification. He is enrolled at three universities, visiting one of the these – the local George Osbourne University (GOU) – every Thursday evening. He finances all of this himself.

On Monday evening, when Alby gets home, he goes straight to his laptop and works through all the notes he has dictated into his smart phone during the day. He has become interested in the evolution of library systems and wants to register this interest on the Research Interest Grid (RIG). While recording notes into his phone, he has also published some of these into StreamingConscious, the latest social network to become popular with researchers, and has gained a few new connections from people with aligned interests, including a promising one with a subject librarian at GOU.

Alby then invokes his Foraging Agent, ‘Bob’. A license for Bob was given to him by a publisher, Coyote, which specialises in resources for software engineers, in return for sending him a steady stream of advertisements. Alby adopted Bob because he liked its interface, but he suspects it has in-built biases towards certain, commercial information sources. He believes that he compensates for this by carefully defining his research questions in Research Question Format (RQF) and filtering the results.

Bob runs constantly on Alby’s ‘slice’ – a portion of Personal Cloud (PC) infrastructure provided by a well-known supermarket chain. After a series of questions and answers, Bob is armed with three carefully RQF research questions, and a set of parameters, such as when to report back, and how much of Alby’s research budget to spend on a single transaction before asking him for approval. Bob has learned through observation how Alby likes to work. It knows him in a sense, enough to represent his interests when dealing with other agents. Alby then instructs Bob to begin searching, negotiating and shopping for answers, leads and recommendations, while he gets on with some reading. Alby has grown to trust Bob.

The Librarian

Charlotte is a subject librarian with many years’ experience (she tried to retire 3 year ago but has been forced to come back to work), specialising in software & systems engineering, and currently working for George Osbourne University. On Tuesday morning she checks the reports from her Listening Agents over breakfast. She controls several agents running on the library’s slice of the GOU cloud.

Bob, an agent representing someone called Alby has made contact, coincidentally, with two of her agents – one which represents GOU and which reports to her, and her own personal agent, Eva. Only yesterday, BirdSong (a social network monitoring agent) had suggested that she connect with @alby on StreamingConscious based on their mutual interest in the history of LMS systems. Charlotte’s interest in LMS systems is partly fuelled by nostalgia – she has been working with such systems for more than thirty years.

She sees that Dan, the GOU agent, has supplied Bob with material to which Alby is automatically entitled, and has automatically reserved two books from the local GOU collection for him. In so doing, Dan recommends to Charlotte the purchase of a newer edition of one of these textbooks.

Dan has also made a number of offers to Bob of more restricted material which can be supplied at a cost, including 3 inter-library-loans. Bob has accepted one of these paid-for items on Alby’s behalf and Charlotte is happy to see that it has also observed the protocol of explaining why it has not accepted the others. In one case, she sees that Bob was successful in bidding on eBay for a second-hand copy of a book which Dan had offered as an ILL. Bob has also made an offer to Dan for ownership of the book, once Alby has finished with it, in return for one free ILL. Dan needs Charlotte to approve this. However, she declines, knowing the book to be flawed, despite its 4 star popularity rating. Dan registers this decision, quietly blacklists the book against any future recommendation, and reports this decision to Bob.

Dan notes that Bob has also registered a second book on Alby’s personal virtual book-shelf and indicated a willingness to make this available to the GOU circulation agent for loan to other GOU students as part of the ‘Support Your Library’ protocol, in return for one free ILL token. Charlotte accepts this offer.

Charlotte instructs Dan to negotiate with Bob to arrange a meeting over coffee for Alby and herself. She does this partly because Eva has separately registered Alby’s interest on the RIG and it seems worthwhile meeting with Alby in person to discuss his research. She decides to investigate a couple of other suggestions thrown up by Dan in the meantime. She also notes that Dan has suggested a couple of other contacts to Bob – other people who are enrolled at GOU and whom Alby may wish to befriend on StreamingConscious – – as part of a strategy to reinforce the local GOU social network of students and researchers.

The student researcher (2)

Later on Tuesday morning, Alby wakes to find an interesting report from Bob waiting for him. He discovers he is the proud owner of a new book on LMS system design and is pleased to note that it has a four star rating – one star above the threshold he has set in Bob’s book-buying decision parameters.

Bob has, inevitably, also turned up a few offers of information and resources from the ‘invisible market’. He knows that if you have the right connections, you can get just about any book in ePub5 format. The penalties for possession of an illegally obtained, copyright resource are stiff however. Although it is not illegal, he is also a little wary of using Turpin, the global federation of Open Access papers and other resources, as he has been culturally conditioned to be suspicious of things which appear to be ‘free’.

He also finds a tentative appointment in his diary for coffee with @charlotte, the subject librarian with whom he connected yesterday on StreamingConscious. As he works close by the university, he accepts the appointment. He can pick up his reservations while he’s there.

Face to face, later that morning

Alby finally puts his pen down, and takes a swig of his coffee. He has been writing furiously for half an hour. Charlotte has just taken him on a whirlwind tour of the evolution of the LMS.

She has described how the library has learned, over the last decade, that client relationship management (CRM) is crucial to its mission. Adjusting to the new realities of social networking and global search, the LMS has become a distributed and loosely-coupled collection of processes, all designed to help connect people with resources and with each other.

Alby learns how the rapid introduction of semi-autonomous software agents into research practice took many by surprise. Although the concepts were not new, and much of the technology existed in one form or another, it took the confluence of a number of factors to finally introduce agent-mediated research:

• the cultural acceptance of an ‘always online’ culture brought about through the ubiquity of smart phones, the prevalence of global social networks and move from the desktop to cloud-based processes
• the utter complexity of negotiating through ‘permission stacks’ to determine whether or not an individual has the rights to access a given resource in a given context
• the complexity of relationships between individuals and institutions

Charlotte explains how, from having been a destination for local researchers, the LMS has dissolved into the fabric of a vast, distributed network of research interests, library collections, national, private and open resources.

While the curation of local collections remains important, the facilitation of networking, and the handling of transactions, both social and financial, has taken over as the focus of the LMS. She points out that where once it was quite easy to point to the LMS – at least as a line in a budget sheet – it has become somewhat nebulous on recent years. The LMS has become the coffee-shop of cyberspace, where software agents meet to compare notes, register interests, make deals….

Taking a sip of her peppermint tea, Charlotte sighs as she remembers how simple it all once was.

A recent post on Blogging Practices Session at the JISC MRD Launch Event (#jiscmrd) contains access to the slides hosted on Slideshare used at the JISC MRD Programme Launch Meeting. In the talk I reflected on the discussion on Should Projects Be Required To Have Blogs? which took place initially on Twitter and then on this blog in February 2009.

The context to the discussion was described by Amber Thomas: “I should clarify that my colleagues and I were thinking of mandating blogs for a specific set of projects not across all our programmes“. During the discussion the consensus seemed to be that we should encourage a culture of openness rather than mandate a particular technology such as blogs. One dissenting voice was Owen Stephens who commented “I note that Brian omitted one of my later tweets – not sure if this was by mistake or deliberately because he recognised it for a slightly more light-hearted comment “i say mandate – let them write blogs!” – but I wasn’t entirely joking.“

Owen’s view is now becoming more widely accepted across the JISC development environment with a number of programmes, including the recently established JISC Managing Research Data and the open JISC OER Rapid Innovation call both requiring funded projects to provide blogs. This current call (available in MS Word and PDF formats) states that:

In keeping with the size of the grants and short duration of the projects, the bidding process is lightweight (see the Bid Form) and the reporting process will be blog-based

and goes on to state that:

We would also expect to see projects making use of various media for dissemination and engagement with subject and OER communities, including via project blogs and twitter (tag: ukoer)

I’m pleased that JISC have formalised this requirement as I feel that blogs can help to embed open working practices in development activities as well as providing access to information which is more easily integrated into other systems and viewed on variety of devices than formats normally used for reporting purposes.

But how should projects go about measuring the effectiveness of their blogging processes and should should the findings we made openly available, as part of the open practices which projects may be being encouraged to adopt, and as data which is available under an appropriate open data – as we might expect data associated with these two programmes in particular – which is unencumbered by licencing restrictions which may be imposed by publishers or other content owners?

Openness for Blog Usage Data

In addition to providing project blogs there may be a need to be able to demonstrate the value of project blogs. And as well as the individual blogs, programme managers may wish to be able to demonstrate the value of the aggregation of blogs. But how might this be done?

A simple approach would be to publish a public usage icon on the blog. As well as providing usage statistics such tools should also be able to provide answers to questions such as “Has IE6 gone yet?” and “What proportion of visitors use a mobile device?“. But beyond the tools which we will be familiar with in the context of traditional Web sites there may be a need to be able to measures aspects which are of particular relevance to blogs, such as comments posted on blogs and links to blogs posted from elsewhere.

A post on Blog Analytic Services for JISC MRD Project Blogs explored this issue and described how tools such as Technorati and eBuzzing may provide lightweight solutions which may help to provide a better understanding of a blog’s engagement across the blogosphere. It should be acknowledged that such tools do have limitations and can be ‘gamed’. However in some circumstances they may help to identify examples of good practice. In addition gaining an understanding of the strengths and weaknesses of such analytic tools may be helpful if the altmetrics initiative which, in its manifesto, describes how “the growth of new, online scholarly tools allows us to make new filters; these altmetrics reflect the broad, rapid impact of scholarship in this burgeoning ecosystem” and goes on to “call for more tools and research based on altmetrics“.

In a post The OER Turn (which is, according to the author, ” the most read post of 2011 on [the JISC Digital Infrastructure] team blog“) Amber Thomas reflects on developments in the Open Educational Resources environment and describes how she now “find[s] [her]self asking what the “Open” in Open Content means” and concludes by asking “What questions should be asking about open content?“.

My contribution to the discussion is that I propose that when adopting open practices, one should be willing to provide open accesses to usage data associated with the practices.

This was an idea I explored in a post on Numbers Matter: Let’s Provide Open Access to Usage Data and Not Just Research Papers in which I highlighted the comment published in JISC-funded report on Splashes and Ripples: Synthesizing the Evidence on the Impacts of Digital Resources which said that:

Being able to demonstrate your impact numerically can be a means of convincing others to visit your resource, and thus increase the resource’s future impact. For instance, the amount of traffic and size of iTunesU featured prominently in early press reports.

which suggests how quantitative data can be used to support marketing activities. But beyond such marketing considerations, shouldn’t those who believe in the provision of open content and who, in addition, wish to minimise limitations on how the content can be reused (by removing non-commercial and share-alike restrictions from Creative Commons licences, for example) also be willing to make usage statistics similarly freely available? And to argue that “my use case is unique and usage statistics won’t provide the nuanced understanding which is needed” is little different from those who wish to keep strict control on their data?

In other words, what is the limit to the mantra “set your data free“? Does this include setting your usage data free?

The quality assurance approaches or metadata we described in the papers was focussed primarily on the service providers. However, six years later, the importance of the quality of metadata for resource discovery is no longer just of relevance to service providers. In a Web 2.0 environment in which content providers can make their teaching and learning and research outputs available on a wide range of services without the mediation of information professionals there is a need to ensure that a wider range of content providers are aware of risks that poor quality metadata can lead to valuable content being difficult to find.

I became aware of such risks while Surveying Russell Group University Use of Google Scholar Citations which I described in a recent blog post.  As mentioned in the post I became aware of the dangers of over-counting the numbers of researchers who have claimed a profile by aggregating researchers from the University of Birmingham with those from the University of Birmingham at Alabama or those from Newcastle University with Newcastle University,  New South Wales.

 Of further investigation I discovered entries from researchers who had misspelt the name of their university by using “univeristy” – a common typo which I myself have made. Currently it seems there are only 33 such misspellings.

In recent years, JISC has invested in some development programmes based around an approach labelled Rapid Innovation. Rapid Innovation, in this context, described an approach of investment in small, short, cheap development projects designed to ‘scratch an itch’. There was more than an echo of the Agile Manifesto in this approach. The Rapid Innovation projects tended to show the following characteristics:

• they brought developers more to the fore
• they produced lighter, more frequent documentation
• they produced working code very early in the process
• they involved end-users directly, and throughout the project

The early work of DevCSI has been informed by this work – notably in the increased awareness of adoption of agile development methodologies.

So why is this important?

The radical changes currently being introduced to the economic and political landscape around higher education in the UK are forcing universities and colleges to re-examine themselves as ‘businesses’. With the growing interest in commodified hardware and software and remote software as a service (SaaS) options for service delivery, HEIs need to examine how they can best exploit these opportunities. (The JISC’s Flexible Service Delivery Programme has been established to help institutions in this). While HEIs will have differing levels and types of interest in what are being referred to as cloud services, they are generally going to be searching for efficiency-based savings.

The value proposition of financial cost-reduction from using shared services is something which cannot be ignored by HEIs – but it seems to me that there are some things which need to be born in mind:

• the biggest saving in cloud-based services is to the supplier, not the customer (although the supplier will pass on some of this saving)
• this whole approach is not yet well understood – especially how SaaS sits with an ‘enterprise’ service oriented architecture (SOA) approach which is also of interest to some HEIs
• some services can be outsourced more easily, or to greater benefit, than others

In The role of the central IT Services organisation in a Web 2.0 world, Joe Nicholls and David I Harrison introduce the useful characterisation of services being either chore or core. Making use of SaaS is a form of outsourcing, and outsourcing is a tricky thing to get right. There are arguments for outsourcing those things you have to do but have no special interest in (e.g. HEIs frequently outsource their catering operation). In the ICT service context such services might include the various administration systems which all HEIs need to operate (e.g. finance). These we might call chore services. However, another reason for outsourcing is a lack of capacity or expertise to deliver a service internally – whether or not that is the preferred option. Services which are core to the HEI’s business might fall into this category occasionally – even if this is not ideal. In a recession, with drastically reduced funding, HEIs might see more core services become unsustainable – or indeed need to reconsider what is core in the first place. Normally, business decisions of this sort are not so simply binary, and some complex judgement will need to be made.

Inevitably, the growing opportunity for outsourcing ICT services will be appealing to many HEIs – whether those services are outsourced to generic or specialist commercial suppliers, or to HE-sector-based consortia such as the Kuali Foundation. But outsourcing can introduce hidden costs. A lessening of control is one obvious concern. But a more insidious risk introduced by an enthusiastic embracing of outsourcing services is a temptation to start to regard the maintenance of local development expertise as a luxury. After all, if we’re going to outsource our ICT, why do we need to retain technical staff and, especially, developers. ICT is just a  commodity, right?

Well, no. I think it is a mistake to lose sight of the advantages that come from a local capacity to perform and deal with technical innovation. A local or ‘in house’ development capacity is a valuable resource in the normal run of things. In a recession, it is vital.

The successful organisation will use a recession to examine its business and to change in order to be ready to fully exploit the economic recovery, when it comes. And large organisations are getting better at preparing themselves to be able to innovate internally or locally. Scott Anthony, who has worked with Clayton Christenson who coined the expression “disruptive innovation”, lists some principles which inform an organisation’s ability to engage in innovation:

• Put the customer, and their important, unsatisfied job-to-be-done at the center of the innovation equation
• Embrace the power of simplicity, convenience, and affordability
• Create organizational space for disruptive growth businesses
• Consider innovation levers beyond features and functions
• Become world class at testing, iterating and adjusting

(I’m not entirely enamoured of the ‘disruptive innovation’ label – as my colleague Brian Kelly pointed out at the recent CETIS Conference, the HEI sector is receiving plenty of ‘disruption’ right now from political forces – certainly enough to encourage innovation!)

In Whither Innovation, Adam Cooper of CETIS asks: “Could we leave innovation to the commercial sector and buy it in?”. Answering his own question, he quotes Cohen and Levinthal (1990) who introduce the term absorptive capacity, describing :

…a model of firm investment in research and development (R&D), in which R&D contributes to a firm’s absorptive capacity….

I see a direct parallel between outsourcing too much, and losing the absorptive capacity necessary to respond to change and to innovate to meet new challenges. In my talk to the JISC Innovation Group, I presented this diagram:

This diagram tries to express the role of the local developer to act as an agent enabling and supporting change in an HEI. The developer deals with the remote, outsourced ICT system at a technical level, becoming one route through which the HEI ensures it gets the best possible value out of this arrangement. Remote services are, nowadays, guaranteed to offer some sort of application programming interface (API) which allows the more technically capable customer to tailor the service to their needs, rather than simply being obliged to use an undifferentiated, default user-interface for example. Local developers are increasingly networked with their peers in other HEIs (not least because of the efforts of the DevCSI project), so they become quite powerful in being able to exploit commonly used remote services through the free sharing of knowledge, technique and even code. And because local developers are, in some case, embracing a more agile approach to development, they become the conduit through which the end-user expresses their needs to make the remote, shared service better fit their local, idiosyncratic needs. Developers can become surprisingly aware of ‘business’ processes and information flows through an HEI, as they have to deal with them at several levels (I wrote about experiences of this sort in a previous post, SOA and reusable knowledge).

I see an opportunity for the DevCSI project to focus its efforts on this aspect of change within our HEIs. Change management is going to be crucial for HEIs as they redefine what is core and what is chore, as they decide what they can do best, and what can be best done for them by others. They are going to need a capable, knowledgeable and above all agile capacity to innovate to meet new business challenges and a changed ICT environment.

I’ve taken to using the label responsive innovation to describe the act of dealing with or instigating technical change in a manner which advances the core mission of the institution. Developers are not the only part of the solution, but they are a vital part. Not only do HEIs need to hang on to their best developers, they need to invest in them, if they are to manage change and not be managed by the changes being imposed on them.

Developers are core.

After careful consideration, I abandoned XForms and threw my energy into putting Rails on top of Fedora Repositories, which I reported on one year later at Open Repositories 2008 in a presentation called How we Integrated Fedora into Ruby on Rails and How You Can Use It.  This in turn lead to the development of ActiveFedora and the resulting Hydra framework.

I was not alone in my decision to abandon XForms.  In January 2008, the W3C published the first draft of HTML5. Later, the funding for the Mozilla XForms project evaporated.  Since then, to my knowledge, none of the browser manufacturers have commited any resources to supporting XForms.  To contrast, all of the major browser manufacturers have committed to support HTML5 (with HTML5 Forms) and have already made substantial headway in that direction.

Meanwhile, my old presentation from 2007 has been floating around the web.  I periodically get emails from people who are enthusiastic about using XForms with their Fedora repositories.  Every time that happens, I do my best to discourage them and promise to put up a blog post enumerating my criticisms of XForms. Thus, with sincere apologies for the lengthy delay, here is the quick rundown of my thoughts on XForms and a bit of info about how MediaShelf is dealing with complex, hierarchical XML in the repository-driven applications that we build.

XForms are to Beta Cassetes as HTML5 Forms are to VHS

XHTML2 and, by extension, XForms have lost their bid to become the chosen replacements for HTML4. HTML5, CSS3 and HTML5’s Forms (formerly WebForms) are the official successors, and they are looming large on the horizon. All of the major browsers already support them.   The further you sink your legs into XForms, the harder it will be to keep up when this wave hits the mainstream.

In case that’s not ’nuff said, let’s look a little more deeply.

XForms is a dead technology.

Technologies are like languages — Latin is a dead language but is still used (and useful) in rare cases. The same is true for XForms.  If you choose to use XForms, you’re choosing to use a technology that will never enjoy broad adoption and will increasingly fall short of contemporary development practices.

Losing the battle to become a standard doesn’t always kill a technology, but in this case it did. XForms was dead in the water in 2008 when the W3C decided to favor HTML5 over XHTML2. The writing was on the wall when the major corporate support for XForms quickly evaporated (ie. IBM cancelled its funding for the Mozilla XForms Project, which to my knowledge has never seen any work since).

While some organizations do still use XForms, this doesn’t mean that the technology is strong — after all, plenty of hefty corporations still run and spend money on maintaining cobol apps.

XForms makes it difficult to stay abreast of contemporary interface and interaction design.

The structure and conceptual framework of XForms feels strange and overly complex to most developers (especially developers who don’t like declarative languages). This discourages innovation on the part of those developers, who are only going to work with XForms when they are forced to. Since XForms effectively died in 2008, none of the innovations since 2007 have happened in the XForms arena. Case in point: XForms is incompatible with the vast majority of the javascript widgets and plugins floating around out there, which in turn means that unless you re-implement all of those plugins you’re stuck with kludgy, outdated interactivity in your forms.

XForms has some real design flaws

XForms assumes that In-Browser interactivity means Asynchronous XML.

With its rigid fixation on XML, XForms prevents you from taking advantage of non-XML communications.

The very notion of AJAX has evolved substantially since the inception of XForms. In fact, the very name AJAX has fallen from favor because what originally manifested as “Asynchronous Javascript and XML” has now become much more open ended. Notably, convention has drifted heavily in the direction of preferring JSON over XML by virtue of its simplicity and attendant ease of serialization/deserialization. Meanwhile XForms instead focuses on embracing the full complexity of XML, which adds complexity without much, or any, real payoff.  Further, with the arrival of HTML5, the predominant conventions, which are not all XML-centric, will become ensconced as the standard.

XForms encourages you to break the MVC pattern in your applications.

Though XForms uses an MVC model internally, it encourages you to break the MVC model in your application by forcing you to embed decisions about your XML schema(s) — which are an implementation detail of your model — into the XForms.  Thus, you’re forced to put representations of your model into the view layer of your application.  This is especially noticeable when your user wants to edit metadata that is actually spread across multiple XML documents (and schemas) in your model — though you can technically handle this in XForms with XInclude, etc, it is far from ideal.

While this is merely bad design when you’ve got just one form in one application, it compounds upon itself as you add new forms because decisions about the schema become replicated across numerous XForm controls within each of your edit views.  Any time you want to change something about your model (ie. what attribute to use on a particular metadata field), you have to update all of the corresponding references in all of your forms.

You could add a layer of abstraction in order to at least partially achieve this separation of concerns, but in that case, once you’ve added an abstraction layer, why the heck would you use XForms instead of a regular dynamic form?

How, then, to deal with complex XML?

This brings us back to the real issue at hand.  Most of the people who ask me about XForms are building interfaces for creating and/or editing complex XML metadata.  Most often they are either dealing with MODS or Qualified Dublin Core.  Since the metadata is already in XML, XForms seem like a perfect fit.

Though I was able to offer criticisms, until recently I haven’t been able to offer a complete, viable alternative to XForms.  ActiveFedora has supported Dublin Core since late 2008, but that was neither a complete nor a freestanding solution, and it wasn’t any help at all for those poor souls who are coping with MODS.  Now, as of last month, this has all changed.

As part MediaShelf’s work on Hydra for the Stanford Libraries, we have created a Ruby gem called OM (Opinionated Metadata). OM is a server-side solution for mapping between your application’s metadata vocabulary and a specific XML structure.  In order to use it, all you need to do is provide a single file specifying how your application terms map to nodes in an xml document.  Based on that information, it uses XPath and some of the characteristics of XUpdate to give you simple tools for predictably creating, retrieving, updating and deleting nodes within complex XML documents.

Though it’s only in its first stages of development, I think OM has some real promise.  We are currently using it in Hydrangea (the Hydra demo application) to edit MODS descriptive metadata and to edit rights metadata based on the Hydra rightsMetadata schema.  Also, ActiveFedora natively supports using OM to index complex, hierarchical XML into solr, so a single OM mapping file gets you all of the basics you need in order to use complex XML metadata in all of your application’s Search/View/Edit functionality while putting regular HTML (either HTML4 or HTML5) into your views.

There is much more information to come on OM.  In the meantime, go directly to the code.

The OM code: http://github.com/mediashelf/om

The Hydrangea code, which uses OM: http://github.com/projecthydra/hydrangea

Within Hydrangea, check out the OM mapping files for MODS Articles and Hydra Rights Metadata.  (Note: Both files also specify custom methods that augment the behaviors provided by OM.)  You can see how the resulting mappings (ie. [:person, :first_name] ) are used in the edit form templates for hydrangea articles.

OpenRepositories 2010 Presentation Proposal (Long Version)

Any repository solution provides facilities for Creation, Management, & Editing of Content as well as facilities for Searching & Browsing through that content. Experience has shown that when a solution binds these two areas of functionality together too tightly, the system becomes brittle and unworkable, discouraging innovation. Our work on the Hydra project has produced a flexible and intuitive solution that combines these two areas in an almost entirely decoupled fashion. This solution, which is already working in multiple Hydra applications, is built on a three-part pattern where Blacklight handles Search & Discovery, ActiveFedora handles Creation, Management and Editing of Content, and a small application called Shelver supplies the crossover point by indexing the content into Solr so that it will show up in Blacklight. This three-part approach reflects a strong pattern for designing and/or improving repository solutions. The main pivot of this approach is to treat indexing as its own separate part of the application and to allow that indexing processes to evolve constantly as part of the application development cycle.

This work is the product of combining established best practices, best of breed software, and lessons learned from an iterative approach to application development. While our implementation is focused on Fedora Repositories, the software could be used in multiple contexts and the pattern is certainly applicable to any content-oriented application.

The anatomy of a Hydra Application

Note: This is a working model of the functional structure of a Hydra application. The complete designs for the final features and functionality of Hydra applications reach far beyond what is presented here. For more information on the greater vision around the Hydra project, please refer to the Hydra Project pages on the Fedora Commons wiki.

• The portion of a Hydra application that handles Creation, Management, & Editing of content is provided by the Hydra Core, which consists of ActiveFedora along with a few Hydra “helpers” which integrate ActiveFedora into Ruby on Rails.
• The Search & Discovery portion of a Hydra application is a Blacklight installation - nothing more, nothing less. As with any Blacklight installation, its behavior and appearance are likely to be customized but otherwise there is nothing Hydra-specific about it.
• Shelver (which can be run either from within the application, from the command line, or as a JMS listener) indexes content and its salient metadata in Solr, usually pulling that content from Fedora.

These three components — Blacklight, and Hydra Core and Shelver — work in concert to present a consolidated repository solution to the end user. Meanwhile, the three components are sufficiently decoupled that each could be run as a freestanding application. They interoperate based on a minimal contract that revolves around decisions about what information should be in Solr and how it should be represented in the Solr index in order to achieve the ideal search experience.

In the process of customizing or extending a Hydra application, some changes require modifications to all three components, but most changes impact only one or two of the components at a time. This makes it very easy to iteratively improve the application and adapt to real world needs.

This structure grew naturally out of a process of exploration. In early 2009 developers at UVA and Stanford discovered that it was relatively easy to put Blacklight on top of a Solr index that had in turn been been populated by ActiveFedora — effectively turning Blacklight into a search & discovery interface for that Fedora repository. Based on that, we tried dropping ActiveFedora-driven views & controllers for editing Fedora content into the same Ruby on Rails application as an existing copy of Blacklight. It worked like a charm. The two systems happily coexist. What we found was that as long as we could change and refine how the metadata percolates from Fedora into Solr, getting Blacklight to operate together with the ActiveFedora management component was completely straightforward.

With most Hydra applications, all content is stored in a Fedora Repository. However, there is nothing to prevent you from adding non-Fedora content to solr and having it show up in the (Blacklight) search & discovery views. Of course, that content will not be editable unless you implement the code to integrate with that content’s host system.

Best of Breed: Blacklight & ActiveFedora

Blacklight is a next generation Search & Discovery tool. It was intentionally designed to serve a single purpose - Search & Discovery - without having any knowledge of indexing, cataloging, or even the location of the content it’s searching through. Whatever information you have in your Solr index, Blacklight will help you expose a rich, faceted search interface for exploring through that information and displaying detail views of individual records. This open-ended design made it very easy for us to integrate Blacklight directly into our Hydra applications as-is. The ease with which we achieved this seamless integration is a testament to the quality of Blacklight’s design.

ActiveFedora is a Ruby library that encapsulates the details of interacting with a Fedora repository and provides high-level tools for defining data models, creating Fedora objects, and modifying the data associated with those objects. While opening the door to rapid, iterative application development, ActiveFedora also attempts to expose and accentuate many of the strong design patterns inherent in Fedora. ActiveFedora’s emphasis on flexibility and design patterns provided us with many opportunities to make our Hydra applications robust and re-usable. In particular, ActiveFedora makes it possible for multiple Hydra (and non-Hydra) applications to operate on top of a single Fedora repository, thus achieving the goal of providing many lightweight views onto complex, heterogeneous repository content.

Hydra Core: the building blocks of an interface for creating & editing Fedora content

Hydra Core provides the Ruby on Rails code that handles Creation, Management, & Editing of Fedora content. This primarily consists of Rails helpers for generating edit interfaces and Rails controllers to handle the submissions from those interfaces.

Fedora allows a great amount of freedom with respect to data models and metadata. As a result, we could not simply create a single generic content management interface in Hydra. Instead, we created a number of “helpers” that allow you to deal with your Fedora content and its metadata at a high level of abstraction. For example, the editable_metadata_field helper generates the HTML for displaying an editable version of whichever metadata field you specify. All you have to know is what field you want to display and where it is stored within the object. Everything else is handled for you.

The forms generated by the Hydra helpers need somewhere to submit their data to. This is handled by the Rails controllers provided by Hydra core.

Underneath the helpers and controllers, Hydra Core relies on ActiveFedora to handle connecting with Fedora, modeling Fedora objects & metadata, and performing the basic operations of creation, retrieval, updating and deletion.

Shelver: a script that brought unexpected freedom

When we wrote Shelver, we didn’t anticipate how integral it would become to the application development process. Shelver started out as something extremely simple. A developer at Stanford initially wrote it in order to populate a Solr index with some working data from a Fedora Repository. Over time, as needs arose, we built out the script to be more robust. It soon became apparent how crucial it is to be able to modify and/or augment the behavior of your indexing tool. In most other systems, the indexing tool is either implicit (relational databases) or external to your application and difficult to re-configure (ie. Fedora GSearch). As a result, when working with other systems, discussion of (and changes to) the indexing strategy are kept to a minimum. In contrast, since we had Shelver at our disposal, we found ourselves constantly tweaking it to satisfy new functionality. This ability to tweak our indexing routines gives us radical freedom to explore new features, improve the search experience, and increase the quality of search results.

Eventually, we pulled shelver into the Hydra app itself so that we could trigger it as part of the save/update process, though we retained the hooks for running it as a command line tool as well. We also did this because we found that changes we made to shelver often corresponded to changes in the search interface. Shelver was continually evolving in conjunction with the application, so it made sense to track the code with a single versioning system.

Approaches to Indexing: from RDBMS to Fedora + Shelver

RDBMS (data model & search index combined)

If you rely solely on a Relational Database to drive your application, your data model (the database schema) is also your indexing model — any search oriented changes necessitate changes to your data model. This makes it difficult to refine and extend the search & discovery portion of the application without impacting other areas of functionality.

RDBMS + Solr (separate search index from data without much thought to the conceptual differences)

A number of tools exist for pulling content from a relational database into Solr. This achieves the goal of separating the search index from the data itself, allowing you to have an indexing model separate from your data model. However, often with these systems the indexing methodology remains tightly bound to the data model. This is more of a conceptual stumbling block than a technical one. It’s easy to underestimate the complexity and distinctiveness of indexing for search & discovery. It is not enough to index your data in Solr; you must think differently about how and why you put it there. This “thinking” must be manifest somewhere in the application’s code, ideally separated from the rest of the application.

Fedora + GSearch + Solr (freestanding tool specifically handles indexing)

Fedora is explicitly designed with the idea that you should separate your data model from your indexing solution(s). This allows us to use any variety of content models and metadata schemas to represent our content in Fedora while pulling that information into any number of indexes to suit specific searching needs. The most common indexing approach with Fedora repositories is to use Fedora GSearch to pull Fedora content into a Lucene, Solr or Zebra index. This approach has the benefits of completely separating the data from the index while also providing a freestanding, configurable tool to handle the process of indexing.

GSearch was designed with the goal of enabling 1) full-text searching of Fedora content and 2) indexing of arbitrary XML metadata from Fedora objects. It runs as a web application alongside Fedora, listening for JMS messages or REST API commands telling it to (re)index Fedora objects. The process by which GSearch indexes the content is implemented as a mix of XSLT and Java code.

GSearch establishes the strong best practice of decoupling both the search index and the indexing process from the data itself. This pattern was part of Fedora’s design all along, but thus far GSearch has been the clearest manifestation of it.

Because it was designed specifically to enable full-text indexing using XSL Transformations (XSLT), GSearch operates on the premise that you are transforming the content in order to put it in the index. In a basic system, transformations are sufficient. However, most repository solutions eventually need to actively process the data when indexing it, performing complex actions in order to decide how to populate the search index. Because XSLT does not lend itself to performing such complex processes, you must modify Java code if you want to implement this type of processing in GSearch. Modifying that code has proven daunting for most. Very few projects have taken on the challenge of modifying the GSearch code itself. Those that have modified the code have only done so in minimal and relatively stable ways.

Fedora + Shelver + Solr (allowing the indexing methodology to constantly evolve)

If you want to provide a great search & discovery experience in your application, you must make it easy to iteratively “massage” the indexes. Anyone who manages a Blacklight or VuFind installation on top of their ILS (or anyone who participates in Code4Lib) can attest to the fact that in order to achieve a truly successful search & discovery experience you must continually refine the way you index your metadata. Little changes in your indexing methodology can bear tangible results for end-users.

In building SALT, the first Hydra application to combine Blacklight with ActiveFedora, we created Shelver as an alternative to GSearch because we wanted to be able to specify our indexing process in Ruby code and, where possible, we wanted to use simple mapping files rather than being forced to use XSLT and Java to perform those actions. We assumed that Shelver would be a relatively simple application whose code rarely changed. After all, when using GSearch we rarely changed the XSLT and basically never changed the Java code. We expected that the same would be true with Shelver. We were wrong. Shelver is constantly changing because we are constantly coming up with new things that we want to do to improve the search & discovery utilities in our Hydra applications. As time passes, the code of Shelver itself has stabilized, but the instructions for how to index specific data from Fedora continually morphs as a regular part of the application development process. In fact, touching the Shelver code has become such an integral part of our work that we can’t imagine building a repository solution without this kind of freedom.

Conclusion, Observations and Best Practices

To review, some of the recommendations coming out of this work are to

• use indexing as the crossover point between decoupled solutions for searching through and managing your content
• make the indexer an explicit, evolving part of your application
• use flexible components that were designed with iterative development in mind
• re-use established best practices where possible
• combine best of breed solutions for astounding results

We were pleasantly surprised to discover how easy it was to combine Blacklight and ActiveFedora into a single Fedora solution. The three-part pattern that emerged out of this effort, which now constitutes a basic Hydra application, builds on well established practices and serendipitously combines them in a stable, intuitive way. This in turn provides a strong base for us all to carry out a great amount of innovative work in the coming years.

Linked Data was the dominant theme today, starting with the pre-conference Linked Data Meetup which included presentations and workshops from Tom Scott, Patrick Sinclair, and Silver Oliver of the BBC and John Sheridan & Jeni Tennison of data.gov.uk among many, many others.

Lin Clark of DERI ran a Drupal RDFa workshop in the afternoon. She demonstrated RDF Mapping, the RDF SPARQL Proxy, and the RDF SPARQL Endpoint modules and gave a preview of the RDF features to expect in Drupal 7 core.

One session I really wanted, but wasn’t able, to attend was Jim Downing’s Multicore Processing session. Not a terribly interesting title, but he worked through mining linked data from data.gov.uk, using the Clojure library Incanter to find correlations in the data, and finally, parallelizing the data processing.

The dev8D spirit of developer happiness was nowhere more evident than in this year’s new attention to electronics. I sat at a table watching Emma Tonkin build an electronic whiteboard with a nano-projector and Wiimote. Walking over to the projects room, I saw Ben O’Steen working with facial recognition software, Dave Tarrant turning off and dimming lights over the local network, Julian Cheal wiring each room’s entry with an RFID reader, and Ian Ibbotson programming an Arduino. Tomorrow, I’m told, we’ll have someone showing off a RepRap 3D printer.

I find the impromptu meetings as valuable as any of the structured components of dev8D. I sat down with Andy Turner, a researcher at the University of Leeds, who introduced me to GENeSIS, a mad-scientist level modeling and simulation project that I thought was absolutely fascinating. Just as serendipitously, as we wrapped up Day 1 of dev8D at the pub (where else?), I found myself drinking a pint with Scott Wilson and we got to talking about Apache Wookie and HTML 5.

See you tomorrow!

The following diagram is meant to represent a design ‘pattern’ which I have seen often proposed, and sometimes implemented, in the JISC Information Environment (IE) as well as in the wider higher education (HE) sector in general:

It is my belief that readers who have been involved with the IE for some time will recognise this, at least in a general sense, if not in specific cases. In this arrangement, an aggregation of data is presented to the end user, through the development of a user-facing application or service. The user-facing service will in almost all cases be a web-interface, somewhat similar to the ‘portal’ concept of old but in a centralised, single, global deployment. Because it is generally accepted to be desirable to make such data available to other services (in keeping with the larger goal of interoperability through open standards), one or more machine interfaces or so-called APIs, giving access to the ‘backend’ of the system, will be offered. What this design pattern aspires to is a service implemented to be both user-facing service and machine-facing infrastructure component.

However, I contend that this is, in fact, what software engineers might call an anti-pattern. An anti-pattern is a design approach which seems plausible and attractive but which has been shown, with practice to be non-optimal or even counter-productive. It’s a pattern because it keeps coming up, which means it’s worth recording and documenting as such. It’s anti, because, in practice, it’s best avoided….

There is much which is implicit in this pattern, so I will attempt to surface what I believe are some hidden assumptions in a new version of this diagram: this is what this design pattern, once implemented, reveals:

In this second diagram, the orange colouring indicates the parts which actually get built and are supported; the yellow indicates the parts which might get built, but which won’t really be supported as a service – in a sense, this is stuff which is believed to work but actually doesn’t; in the case of the users, the yellow colouring indicates that their demand for this service is believed to exist; those components in the diagram which are neither orange, nor yellow, are the product of little more than speculation. In the end, the investment in creating a user-facing application based on an expectation of future demand which doesn’t materialise is wasted while, at the same time, the investment in providing unused machine interfaces is also wasted.

I believe that this design pattern rests on several assumptions which are actually fallacies, and is, therefore, an anti-pattern.

Fallacy 1: “Build it and they will come”:

While infrastructure services can, indeed should, be developed with future opportunity in mind, it is helpful to have an existing and real demand to satisfy, which the new development addresses. If the service is demonstrably useful to users, and is developed effectively with future opportunity in mind, then there is more chance of the service actually working, and of it being attractive to developers working on future opportunities.

Fallacy 2: Interoperability through additional machine interfaces:

Machine interfaces need as much specification, development, testing an maintenance as user-interfaces. Simply making a machine interface available through the adoption of a platform which has a built-in facility offering some standard interface is not enough. A system which proposes to offer three or four APIs is quite likely not going to support any of them adequately. I have argued before that ‘interoperability is not enough‘: in fact, this arrangement does not often lead to interoperability, let alone actual exploitation of the capability to interoperate.

Fallacy 3: People/organisations who can make good infrastructure are also going to be good at building end-user-facing services (and vice versa):

Effective infrastructure supports services which in turn support end-users. The skills and knowledge required to support service-providers are generally quite different from those needed to deliver good user-facing services.

I call this the infrastructure service anti-pattern because the result comes from conflated requirements to deliver both infrastructure (machine-to-machine interfaces) and compelling user-facing services and applications. The result can be something which satisfies neither requirement. The users, requirements and priorities are often completely different between these two problem spaces. I suggest that the following are some possible reasons for this anti-pattern appearing:

• funding (naturally) tends to follow services, happy users and, importantly, new features.
• funders like to see their investment showcased
• infrastructure is mostly invisible making it hard to ascertain impact from users

Proposals for alternative design patterns

Here is a suggested alternative design-pattern:

In this design pattern, the API is developed before any user-facing application, or at least in parallel. An application is developed to exploit this API based on real users requirements. No service is developed until such requirements can be identified. This means that an API will be developed, and it will be being used in at least one case. Opportunities for third party integration for usage of the service are, ideally, identified beforehand. The API is properly supported from the start, or else the service fails completely. The value proposition being offered for further, opportunistic third-party developments, whether real or imagined, is now real and, crucially, supported.

An interesting alternative to this is the approach of combining the user-facing web pages and the machine-actionable API into one interface, through embedded RDFa for example:

It remains to be seen how this approach is going to work out over time, but we have seen hints of simpler approaches to combining user and machine interfaces in the past, such as RSS being styled to give a decent human-readable interface, or earlier attempts to do interesting things with XHTML.

I wonder if readers agree that the first diagrams represent an anti-pattern which they recognise. And would the proposed alternatives fare any better?

If you have not yet taken a look at CDL's curation micro-service specifications, most of which may be printed on as few as one or two sheets of paper, see the Digital Library Building Blocks.

My co-workers in the LC Repository Development Center have been chatting about these specs on and off throughout the year. After months of procrastinating, I finally read all of the specs on Thursday; it's wonderful that you can do so in the course of one reading session, I might add. Yesterday a bunch of us RDCers got together to chat (informally) about the specs: what they're for, how they work, and how they interact with one another. I learn by doing, by examples, so I combed through each of the specs in advance of our meeting and tried to construct a minimal repository[1] based on micro-services.

Here is a tree visualization of the final product, inevitable warts and all: The services I used were Namaste, Content Access Node (CAN), Pairtree, Dflat, Reverse Directory Deltas (ReDD), Class-based System for Managing Object Properties (CLOP), and BagIt (co-developed by LC and CDL).

As I mentioned in our Friday meeting, recounting my experience exploring the specs: the bad thing is that I spent an hour building a repository with rudimentary tools such as mkdir, touch, cp, ln, and emacs; but the good thing is that I built a repository in one hour using common, rudimentary tools. It's a very compelling paradigm. Ed's already built a tool implementing some of Dflat, further demonstrating how lightweight these micro-services are. (UPDATE: Ed notes that this code is a work in progress and is "barely functional.") (UPDATE 2: The dflat library has come a long way. Check it out if you're interested. Also, I just committed a pretty basic Namaste library: http://github.com/mjgiarlo/namaste. Only took about an hour, which is a testament to the power of lightweight specs.)

I am certain this will be a running thread at work as the specifications evolve and our understanding of them grows. Some questions and comments that occurred to me while exploring the micro-service specs and building the minimal repo:

• CAN was a bit puzzling. The spec is simple enough, but I found some of the conventions confusing, and I was left wondering what CAN provides other than a container. What I would like to see is a simple use case and perhaps more examples. Thus, the CAN stuff in my sample repo doesn't feel very useful only because I had a hard time working with the spec.
• CLOP feels like the least mature of the specifications. It seems generally useful to be able to put digital objects, however you define that, into classes and define properties on those classes. The spec did not clearly convey to me just how it accomplishes that aim. A few examples would go a very long way. I've got some CLOP stuff in the sample repo but I have no idea how close my implementation matches the spec.
• Is Dflat dependent on ReDD? One would assume not since there's an optional property in the dflat-info.txt file for specifying a delta scheme. But, say, could you stub out the v001 directory (reserved to hold the initial version of a digital object) and use a system such as git or bazaar?
  
  One might argue that these established delta schemes, if you want to call them that, have many more developers and users than a system such as ReDD and thus should persist longer and have more tools built around them. I imagine the micro-service viewpoint would acknowledge that point, but counter that the spirit of these specs is to avoid dependencies from outside the filesystem?
• Is the ReDD specification meaningful outside of a Dflat given that any one ReDD directory knows nothing of its successors and predecessors, or is it dependent upon Dflat?
• Could a BagIt bag live inside of the ReDD reserved "full" directory? That is, could the "full" directory be marked up appropriately to be a BagIt bag?
• How many tools exist for these specs? I notice there's code in CPAN for Pairtree and Namaste, which is a fabulous start. Tools are the difference between YAMF (Yet Another Messy Filesystem) and reliably managed curation services. Granted, tools such as cp and emacs already exist and are part of the appeal of these micro-services, but there's also tremendous room for error if operations are all done "by hand."
• To what extent has CDL transitioned to using these specs/tools?
• Are other institutions using these specs/tools? I have heard tell that digital library folks from the University of Michigan and the University of North Texas may be involved.

I hope I don't sound overly critical. I'm really glad our colleagues at the California Digital Library have written these specifications and applied their deep experience to what could be a transformative paradigm[2] in the digital curation world. Kudos to them!

Notes

1. Perhaps it's more in line with the specs to refer to this space as "a managed filesystem that drives repository and curation services," given the CDL philosophy that preservation is not a place/repository. But it's easier to say "repository," so there you go.
2. Please excuse the fanboyishness; this filesystem fetishism is exciting stuff!

The survey closed in July, and the subgroup spent August writing a report on the survey results. That report is now final and it's available to the public. Feedback may be sent to our (woefully underutilized) public i2info mailing list, left as a comment on this post, or e-mailed to me privately which I can forward to our internal list.

The next step is to build upon the report to draw yet more conclusions from the data — there's an awful lot there — and flesh out some repository use cases for institutional identifiers. The I2 core group is moving quickly towards finalizing identifier metadata elements so that a standard may be drafted, and I think having some use cases documented will help drive the standard in a direction the community can get behind.

Onward and upward.