Taking inspiration from Jim Jagielski's opening keynote "Open Source: It’s just not for IT anymore! (pdf)", we wanted to help foster a community around the microservices, and so we took a number of initial steps to convert the various implementations of ruby microservices into a better community-driven, collaborative project:

1. Created a microservices "organization" on github to hold the community-driven source code repositories. Before, the projects were held under a personal account that had a diversity of projects in various states of use and support. By creating a topic-driven organization, we hope to attract contributors and promote easier discovery of these projects
2. Created a mailing list to record decisions, answer questions, and collaborate.
3. Agreed to a set of standards and practices for microservices projects to ensure consistency and quality across these projects:
   1. Basic "meta" files -- like README, TODO, LICENSE, etc -- should be present and contain enough information to help people get started using and contributing to the projects
   2. Clarified source code licenses, and standardized on the Apache Public License 2.0 for each project.
   3. Vastly improved the source code testing and documentation coverage, and standardized around rspec and yard. Projects are now subject to continuous integration to ensure tests pass, documentation is built, and test coverage remains high.

Managing digital media content adds different challenges to file management than traditional text and images. The content is time based, and therefore more complex. Even the metadata needed to describe all aspects of the content to support better access is more complicated. Even after media materials have been cataloged, digitized, and stored in a repository or database, scholars and archivists lack the tools to manage and expose the data to the world. Significant workflow challenges exist to go from large, preservation-quality digital files to media appropriate for delivery across the Internet.

The WGBH Media Library and Archive department (MLA) manages a collection of over 750,000 items dating back to the late 1940’s. As an educational foundation and the creator of a valuable collection of media resources, WGBH has embraced new developments in online media in its efforts to bring its archived materials to a broader audience and to serve the needs of the academic community. WGBH is successful in exposing content for the public through national production websites such as American Experience, FRONTLINE and NOVA, whose customized and carefully constructed features and services create added value for end users by encouraging the dissemination and use of WGBH- owned content. Like many similar institutions, this has been supported by the deployment of many ad-hoc, silo-ed content management systems on a project-by-project basis with each portal maintaining unique metadata and media assets, making it difficult to create new, innovative interfaces and services with the underlying content.

In 2000, in partnership with a vendor, WGBH developed a DAM architecture for media access and published reference architecture documentation for other media organizations to replicate the work. The preservation DAM system is based on a proprietary system from the publishing and creative industries with limitations for metadata structure and interface. The vendor tended to develop the system toward what they saw as market trends and viable business sales. WGBH has found that although the system works, it is not flexible to the changing needs of the media industry, and the vendor is unable to tailor the software to our particular user needs without significant additional investment. In addition, upgrades are costly and time consuming, and all of the site-specific customizations built around the software need simultaneous upgrading by internal teams (e.g. extensive customizations to support media ingestions of large video files requiring limited technical knowledge). The customization links often break and need to be rewritten with every upgrade.

Access

To address the need to expose archival content in a sustainable manner, for a variety of audiences, and to encourage innovation within media archives, WGBH created Open Vault2, which provides a digital access portal into a cross-section of material from the WGBH Media Library and Archives. Although designed as an access portal, a secondary objective in creating Open Vault was to explore the potential for the system to fit within the multifaceted content management ecosystem for both access and preservation use.

WGBH Open Vault is built using Blacklight3, Solr and the Fedora repository. Beyond the Open Vault user interface, we exposed a number of APIs, either for internal use or to support existing data exchange projects, including Atom/RSS feeds, unAPI4, oEmbed5, and OAI-PMH. By taking advantage of existing open-source solutions as much as possible, we were able to focus our efforts towards domain-relevant issues. This has proven a reliable platform, and we have since deployed similar technology for a couple cross-institutional, data-intensive projects.

In 2006, WGBH launched Open Vault, an access repository based on CWIS. This site combined clips of media assets from four different series (three of which had separate finding aid websites created earlier).

In 2008/9, WGBH MLA and Interactive completed an Andrew W. Mellon Foundation funded project which allowed us to work closely with humanities scholars researching their needs and habits in using digital media in their work. We developed a prototype, dubbed "Open Vault Research", using Fedora and a PHP front-end. One discovery was scholars lack tools for working with media, while traditional scholarship is still focused on citing textual resources. To address this, we created a number of tools for working with media material:
- aligned transcripts, which allows the user to rapidly scan the transcript of an interview, and seek immediately to a section of interest;
- annotations + tags, which allows the user to segment and describe media fragments and refer back to those notes later;
- fragment addressing, which allows the savvy user to deep-link into a particular point in an object.

Taking these user needs into account, we developed Open Vault v2 using Blacklight and the Fedora repository.

Finally, we are about to deploy a new iteration of Open Vault using Blacklight 3.0 (and, as a footnote, although our application has significantly different behavior, the customizations are only about 3500 lines of code, more than half as HTML templates). Although the Hydra framework as matured significantly since the beginning of the project, because the management of the media and metadata is still performed in external systems, we continue to access the Fedora APIs directly.

In this redesign, we looked at usage patterns over the collection and re-organized and re-prioritized elements of the user experience.
- The majority of our users entered the website at a record page from an external search engine (with about a 50% bounce rate). However, if a user stayed and watched a video, often they would navigate the website to "related content" (exposed using solr more like this)

- Subject browse was used more frequently than expected to give an overview of the materials in the collection

Re-use

Digital Commonwealth is an on-going project, to which we began contributing material from our first iteration of Open Vault using OAI-PMH.

Project Vietnam was a collaboration with the Columbia Center for New Media, Teaching and Learning, that embedded material from Vietnam: A Television History that we exposed on Open Vault. We spent a significant amount of time figuring out how to exchange media and metadata with CCNMTL and settled on a handful of open standards.

The Mozilla Foundation/WebMadeMovies also wanted to work with us around HTML5-based media experiments. To develop a quick demonstrator, Mozilla, with little assistance or guidance from the Open Vault team, was able to successfully build a javascript-based discovery interface using our OpenSearch API, and integrate both our video content and TEI-encoded transcripts, into their popcorn.js environment.

Technology

For our media player environment, we needed a technology that supports several requirements:

• the ability to jump into any point of an item, which is especially important when serving hour long raw interviews (which excludes standard delivery (over HTTP, or otherwise) of the content),
• an open source, or low cost, delivery platform,
• a robust javascript API that allows us, at a minimum, to programmatically adjust the playhead (which we use to provide media/transcript synchronization, "deep linking" into a video, and annotation of media fragments)

We finally settled on a Flash-based player (which provides a more consistent user experience) with an HTML5-based fallback (to support iOS devices and others).

For delivery, we're using h.264 pseudostreaming, which is delivered over HTTP (and is fully compatible with traditional HTTP delivery for clients that don't support h.264 pseudostreaming, which makes alternate uses easier).

To support ease of reuse, we adopted principle that the “Website is the API”, and in so doing, were able to bake in discoverable standards and approaches that are replicable to other holdings and implementations. This approach included semantic markup, adding additional contextual information (as alternative link relations), and exposing user state information (e.g. the location of the playhead) within the page content.

To support advanced usage, we also expose a number of auto-discoverable APIs which expose structured information to recompose page elements without needing to parse web page HTML.

OAI-PMH

To support "traditional" aggregation, like the Digital Commonwealth project, we have an OAI-PMH endpoint. (see also why OAI-PMH should die)

OpenSearch (blacklight)

For other aggregation efforts, we provide an OpenSearch endpoint that allows simple machine-to-machine discovery in a standard way.

Atom/RSS (blacklight)

All search results expose a discoverable Atom/RSS feed. Blacklight also provides functionality through the Document Extension Framework that allows clients to request specific representations of objects as part of the content of the feed.

unAPI (blacklight plugin)

The unAPI endpoint allows applications to discover structured information based on an identifier and a content type.

oEmbed (blacklight plugin)

oEmbed, rather than forcing implementors to discover media assets (through page scraping or unAPI), allows a client to discover the embeddable properties for an asset (and construct a player) in a standard way. oEmbed provides an easily parseable set of metadata required for embedding and, possibly, a pre- generated player implementation;

HTML <meta> tags

While encouraging re-use of materials, we documented possible improvements to make ad-hoc innovation and mash-up creation significantly easier, including:

• oEmbed, rather than forcing implementors to discover media assets (through page scraping or unAPI), introspect the assets for technical metadata, and then construct a player, oEmbed provides an easily parseable set of metadata required for embedding and, possibly, a pre- generated player implementation;
• additional information in the Atom/RSS feeds, in particular ensuring the data contained within the feed representations is comparable to the normal user interface;
• and, exposing additional information on the page for developer-use, which, in the case of technical or rights metadata, is less relevant to our primary audience, but may be essential to building third-party interfaces to content.

  filter: function(array, term) {
		var matcher = new RegExp( $.ui.autocomplete.escapeRegex(term), "i" );
		return $.grep( array, function(value) {
			return matcher.test( value.label || value.value || value );
		});
	}

While this works, it doesn't provide relevancy ranking or near-matches, both of which are important when selecting from long lists of values, some of which are not well-known or contain a significant number of obscure items.

To address this, I added a custom data source to the Autcomplete widget that uses the LiquidMetal library, which is a refinement of the Quicksilver scoring algorithm.

		source: function(request, response ) {
			var arr;

			if(request.term == "")  {
			return response(data);
			}

			arr = $.map(data, function(value) {
				var score = LiquidMetal.score(value, request.term);
				if(score < 0.5) {
				  return null; // jQuery.map compacts null values
				}
				return { 'value': value, 'score': LiquidMetal.score(value, request.term) };
			});

			arr = arr.sort(function(a,b) { return a['score'] < b['score'] }) ;
		  	return response( $.map(arr, function(value) { return  value['value']; }) );
		}

Surprisingly easy.

I've heard a rumor that trackback urls used to be a standard part of the PBS web site infrastructure years ago -- I'd be very interested to know what, if anything, was learned from collecting that data.

For an archives project at work, we implemented unAPI as a simple way to segment the page content and expose underlying metadata formats, in order to offer our partners a quick (and content agnostic) way to pull elements into existing tools. The API endpoint is exposed within the page header:

<link rel="unapi-server" type="application/xml" title="unAPI" href="http://openvault.wgbh.org/api/unapi/" />

Within the page, any number of unAPI IDs are embedded:

<abbr class="unapi-id" style="display: none" title="org.wgbh.mla:0119be4cad49d0c0f47e9eca1d343e0464539a4c"></abbr>

An unAPI client can request a list of formats from the service:

<?xml version="1.0" encoding="UTF-8"?>
<formats id="org.wgbh.mla:0119be4cad49d0c0f47e9eca1d343e0464539a4c">
  <format type="application/xml" docs="http://www.openarchives.org/OAI/2.0/oai_dc.xsd" name="oai_dc"/>
  <format type="application/xml" docs="" name="pbcore"/>
  <format type="image/jpeg" name="jpeg"/>
</formats>

These formats could be used to share any type of data -- different flavors of metadata, content outside the application context. As with oEmbed, this provides a basic way to provide federated and aggregated data within a common framework. In addition to being a convenient service to share content among applications, there is also support for unAPI within the Zotero citation management tool.

What could this look like in order to access NPR.org content? Instead of a rich API, by simply feeding a story url into the unAPI service, an application could retrieve the different content elements -- a text/html or text/plain representation of the story, the audio/mp3 from the broadcast, an image/jpeg feature image, and perhaps an application/xml+rss feed of the series, comments, or category. It lacks the power behind the stand-alone API, but provides data in a form that is a little easier to craft new ways of highlighting this content on station websites.

oEmbed is a format for allowing an embedded representation of a URL on third party sites. The simple API allows a website to display embedded content (such as photos or videos) when a user posts a link to that resource, without having to parse the resource directly.

oEmbed allows the content provider to expose content from a web page (along with a basic set of metadata to support that), allowing an application to embed content on behalf of a user without assuming the user knows what to do with raw HTML code.

Here's example JSON output from Youtube:

{"provider_url": "http:\/\/www.youtube.com\/",
"title": "NOVA | Emergency Mine Rescue",
 "html": "<object width=\"480\" height=\"295\">[...]<\/object>",
"author_name": "NOVAonline",
"height": 295,
"thumbnail_width": 480,
 "width": 480,
"version": "1.0",
"author_url": "http:\/\/www.youtube.com\/user\/NOVAonline",
"provider_name": "YouTube",
"thumbnail_url": "http:\/\/i3.ytimg.com\/vi\/NUrLEKfHB_0\/hqdefault.jpg",
"type": "video",
"thumbnail_height": 360}

When given a URL, for which there is an oEmbed endpoint defined or discoverable, an application can query the oEmbed service to retrieve the embed code and automatically insert it into the page. The great thing about this standard is aggregating media from any compliant source is now as easy as writing text, with all the heavy lifting done in the background.

---

In preparing this post, I noticed PBS Video is offering the oEmbed discovery endpoint, however the offered URL returns a 404 error rather than embed content. So close. (looks like oEmbed works on some videos and not others..)

I'm hoping this is a practical set of several easy-to-implement standards in public media digital projects, especially as organizations are thinking about ways of building communities, becoming better neighbors, talking about aggregation and decentralization, and more. The standards I'm discussing are used in contexts much large than public media, are inherently useful, and systems-agnostic.

Friend of a Friend (FOAF)

There is currently no centralized, accessible database of public media organizations, and no group who is willing to take on the headache of populating, maintaining and administering such a creation. Fortunately, most organizations (and likely all the salient ones..) have some web presence and we can use the already-decentralized web to model our decentralized organizational structure. From the project website:

The Friend of a Friend (FOAF) project is creating a Web of machine-readable pages describing people, the links between them and the things they create and do; it is a contribution to the linked information system known as the Web. FOAF defines an open, decentralized technology for connecting social Web sites, and the people they describe.

Within the context of public broadcasting, what can FOAF do? (In a gross simplification), if every organization published an authoritative FOAF document, containing any information each station thought relevant, we could link, aggregate and query the decentralized data set to begin to answer any number of questions programmatically (where is the closest NPR station? what is the URL for streaming audio for station XYZ? what is the pledge phone number for every station in Wisconsin?).

Here's a quick demonstration document for a public media station:

<rdf:RDF
      xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
      xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
      xmlns:foaf="http://xmlns.com/foaf/0.1/"
      xmlns:admin="http://webns.net/mvcb/">
<foaf:PersonalProfileDocument rdf:about="">
  <foaf:maker rdf:resource="#wkar"/>
  <foaf:primaryTopic rdf:resource="#wkar"/>
</foaf:PersonalProfileDocument>
<foaf:Organization rdf:ID="wkar">
<foaf:name>WKAR</foaf:name>
<foaf:age>78</foaf:age>
<foaf:mbox rdf:resource="mailto:webmaster@wkar.org"/>
<foaf:phone rdf:resource="tel:+15174329527" />
<foaf:homepage rdf:resource="http://wkar.org"/>
<foaf:weblog rdf:resource="http://www.publicbroadcasting.net/wkar/news.newsmain" />
<foaf:tipjar rdf:resource="http://wkar.org/give/" />
<foaf:tipjar rdf:resource="tel:5174323120x371" />
<foaf:isPrimaryTopicOf rdf:resource="http://en.wikipedia.org/wiki/WKAR_(AM)"/>
<foaf:isPrimaryTopicOf rdf:resource="http://en.wikipedia.org/wiki/WKAR-FM"/>
<foaf:isPrimaryTopicOf rdf:resource="http://en.wikipedia.org/wiki/WKAR-TV"/>
<foaf:depiction rdf:resource="http://wkar.org/images/wkar-w-140x50.gif"/>
<foaf:logo rdf:resource="http://wkar.org/images/wkar-w-140x50.gif"/>
<foaf:member rdf:resource="http://www.mprn.org/foaf.rdf#mprn" />
<foaf:member rdf:resource="http://www.pbs.org/foaf.rdf#pbs" />
<foaf:member rdf:resource="http://www.npr.org/foaf.rdf#npr" />
</foaf:Organization>
</rdf:RDF>

I'm making no claims about the accuracy, correctness or well-formedness of this document, I'm just offering it as an example of what could be done.

Because this format is just an RDF document, it is trivial (and encouraged) to extend the FOAF vocabulary with domain-specific information from other sources, e.g. the BBC Programmes ontology.

FOAF has rules and methods for FOAF document discovery and by creating a mesh of organizations we could assemble a full, real-time picture of public media organizations without the overhead of centralization and contribute to a tiny piece of the larger web of knowledge without significant work on the part of any single individual.

First, and most obvious, I did a little bit of styling work to take it from barely functional to ugly-but-useable. If anyone has design ideas for the site, please leave a comment. Other web galleries usually severely crop their images, but I'm attracted to the full page screenshots, but that comes with a whole range of other design challenges.

As part of the design work, I've moved the "user-generated" content forms to the gallery view (and eliminated the station-view entirely), which I hope will encourage some conversation. I toyed with the idea of integrating with services like twitter and delicious to provide comments and tagging, respectively, but I couldn't come up with a good way to distinguish between conversation and criticism.

I've been working on a couple ideas for a proper home page to the gallery to help give the site some context besides these blog posts. I'll try to make some more progress on this as long as people believe this could be a useful resource for the public media system.

Shortly after the first prototype, I started migrating data into Solr (using the ruby sunspot library) and earlier this week I added full-text search for page content (which may or may not match the screenshot content). I'm still playing with approaches to crawling station websites to extract different types of pages (schedules, contact, news stories, features, etc) with Anemone, hopefully I will have something interesting in the next couple weeks.

I'm still working out ways to automate (and schedule re-occuring) screenshot updates, which is complicated by the lack of decent cross-platform tools. On Mac OS X, I've been using webkit2png, which has been great, but this server is running Debian Linux and the only comparable utility I've found is wkhtmltopdf, which requires a patched version to QT. Messages queues or workflow engines seem like overkill, so in the meantime I'll do manual updates occasionally.

As always, the source code to the site is available at http://github.com/cbeer/publicmediatech-stations for anyone interested in hacking on the gallery or just seeing how it was put together.