How would you describe the current state of OER discovery?

Let me start by saying that six or seven years ago, surveys routinely showed that simply “finding” OER content was the most significant barrier to adoption. For several years after you could count on seeing some mention of “difficulty finding OERs” in articles and reports.

I’m happy to say that in recent years–assuming you stay away from commercial publisher’s brochures–you rarely see issues around OER discovery getting those 30 point headlines.

But what I do still see, three years after we launched our Mason OER Metafinder is that while it’s true we don’t talk so much about simple discovery anymore…it seems we’re all thinking more than ever about the need for more efficient discovery. 

What do I mean by more efficient?

• higher signal to noise ratio in retrieval
• less duplication of content in our search results
• unambiguous usage rights for every item retrieved
• and way to quickly assess the pedagogical “fit”

If we had these last two bits of information–usage rights and pedagogical fit–we could easily slice and dice our result sets via facets.

If we ignore for a moment the traditional discovery solution–wherein the searcher dives in and out of various silos looking for appropriate material–there are two approaches to solving the discovery problem across multiple content providers. 

• use a just-in-case system like MERLOT or SUNY’s OASIS. The “just-in-case” tag comes from supply chain management…you store a lot of inventory just in case someone needs it. Here, the inventory is the metadata that’s harvested from various OER content providers. That metadata is normalized to some degree…then indexed…and it’s that index that you’re searching). Results are limited to items the search system knows about.
• or use a just-in-time system like the Mason OER Metafinder. Ours is a “just in time” system because, again drawing an analogy to supply chain management, we maintain no stock but rely on prompt delivery from our suppliers: OER content silos. Instead of searching a pre-built index, when you submit your query the metafinder launches up to 21 real-time parallel searches across each of the up to 21 sites that you have asked to search. It then collects, dedupes and ranks the top 100 results from each of these sites, combining all into a single, faceted results set.

For now, I’ll conclude by pointing out that each of these approaches has advantages and disadvantages and as you might expect, each poses dramatically different maintenance requirements. To help bridge the gap between these approaches, we include metadata aggregations like MERLOT and OASIS along with content providers as search targets in our Metafinder. 

What are the main challenges/specific needs can you identify at this time?

We have several interesting issues in the OER content world that complicate discovery. 

• First, there’s very little standardization of metadata beyond Author and Title and publication date. And across repositories, even those simple and seemingly straightforward metadata elements tend to drift a bit.
• Then, there’s willful duplication of content across repositories. That redundancy is useful I suppose in a world where repository sustainability is always a concern…but once you open up cross-repository searching, it poses complications. For example, looking at results in the OER Metafinder, I’ve noticed that sometimes the same content is in two or more two or more repositories but there will be a slight variations in the author/title metadata on each site. Hard to teach a machine to unravel that duplication or how to select the most appropriate copy.

What approach(es) do you think would best address these needs?

So from my vantage point–which is trying to offer a search engine that increases search efficiency – the key to fixing many of these issues is standardizing on a particular metadata schema for OER content…and then devoting time to enriching that descriptive metadata.

If I could just issue a decree, it would be that the community settle on a metadata schema that suits at least the basic needs of all interested parties. By that I mean let’s not follow our natural librarian impulse to over-engineer the solution before we deploy it but let’s focus on figuring out the minimum that improves on the current state of affairs but also offers an extensible design that can evolve and improve as we work with it. That simplicity will also speed adoption of the schema.

Then I’d give preference to those repositories and content providers that utilize the schema. 

What, if any, success stories do you know of?

I think one success I’ve noticed is the growing worldwide reach of OERs and the inter-connectedness of the world when it comes to OER discovery.

I try to track any library, libguide or webpage that provides a searchbox or link to our OER Metafinder. I post a list of those sites on our “About the Metafinder” page…and from the list link back to the page that links to our service. What that has turned into is a quick spot to view more than 400 OER-related libguides, websites or services. More than once I’ve heard from people who appreciate being able to so quickly find OER advocacy materials from literally around the world. For example, this past month, among the top 25 sites sending traffic to our Metafinder were sites in South Africa, Australia, Canada, Kenya and Taiwan and the Netherlands.

How can we work to reduce silos in OER discovery initiatives?

In the absence of any sort of cross-repository search mechanism, it’s absolutely true that the problem of discovery becomes more difficult as the number of silos increases. There’s a limit to a searcher’s energy and diving in and out of silos all day can be exhausting. If, however, we have a more standardized way of surfacing the relevant content of each silo then the number of them is more a computer scaling issue than it is a burden to searchers. So we’re at something of a fork in the discovery road. Do we think about how to reduce the number of places you have to look or do we think about how we might build a single virtual OER database out of the many siloed repositories.

What role, if any, does accessibility and equitable data play in OER discovery?

Equitable data or Data Equity, is all about finding and eliminating the ways that bias, assumptions, unfairness and prejudice can slip into a data project. I suppose you can find traces of those problematic impulses somewhere in the OER discovery universe but my sense of things is that the OER movement in general is already quite far ahead of many other activities in valuing the open and equitable. So are there areas where we can improve? I can think of one, and that would be making sure bias and prejudices are not reflected in the metadata we develop in hopes of improving discovery. Thinking about equitable access, I think we might also work to insure that our discovery platforms and our content delivery sources support the simplest, least-expensive device capable of reasonable function–rather than requiring an expensive computer to enjoy the best experience.

The Emergency Temporary Access Service provided to qualifying HathiTrust members has proven quite popular with George Mason University users.  Here’s a chart showing item circulation by month, beginning April, 2020 and running through March, 2021.

Graph covers activity between July 3rd and December 9, 2019.

This reflects only the DNS query activity on the campus network.  Mason affiliates using off-campus networks are not included in this chart.

We can assume, I think, that most activity going to SCI-HUB is all about finding content.   ResearchGate also fills this role but as a social networking platform, there are other reasons for traffic to their site.   We can see that ResearchGate and Google Scholar are heavily visited.   What we can’t readily see is the degree to which they serve as an alternative source for otherwise restricted content.

You can download our dataset here (roughly 8 Mb):    https://tinybox.gmu.edu/files/DNS-QUERY-Final.txt

But as the cost these bundles has risen–far outpacing inflation–libraries have begun looking for alternatives. Some are letting their “big deals” expire while others are developing strategies to help inform those looming (and often fraught) renewal decisions.

SPARC (the Scholarly Publishing and Academic Resources Coalition) has been carefully tracking this activity and their work provides an easy way to keep up-to-date on most aspects of this issue.

But one question I’ve had for some time is what sort of gravitational pull are sites like Sci-Hub or ResearchGate exerting on the already disrupted orbits of users, libraries and publishers?  Put another way, if researchers are satisfying their content needs outside the library/publisher channel, shouldn’t we factor that into our strategy around these big deals?

I realize I’m not the first to ask who’s using Sci-Hub.  Here are just a few of the many articles that get at this topic:

• Who’s downloading pirated papers? EVERYONE (Science, April 2016)
• Sci-Hub provides access to nearly all scholarly literature (eLife, March 2018)
• Shadow Libraries and You: Sci-Hub Usage and the Future of ILL (ACRL Conf Paper, 2017)

Each talks about usage activity and traffic patterns but in a way that is little more than anecdotal background noise if you’re trying to fashion a local strategy and need to focus on what your local users are actually doing.  Simply asking who’s using these sites poses all sorts of problems.

I finally settled on analyzing DNS queries to our campus nameservers as a reasonable metric.  When a user on our campus network points his browser at researchgate.net, our campus nameserver logs the transaction.  An imperfect measure to be sure (e.g., it ignores traffic to “shady” sites from off-campus affiliates using their ISP’s nameserver) but it does let me compare on-campus traffic to “pirate” sites with on-campus traffic to sites provided via our library’s subscriptions.

Mindful of privacy issues, I asked a friend in campus IT to take a list of 6 or 7 domains and derive an extract file from the DNS query logs, providing just date, time and query string for anything that matched the domain information I provided.  Here’s an excerpt of the result:

Producing this extract is now part of a weekly cron job so I’ll be able to monitor the relative use of these sites over the coming months.  In this one particular instance, I can’t wait for the Fall term to begin…  [ update:  You can see subsequent months here ]

So what did I find by monitoring DNS queries between July 3rd and July 13th?

The graph shows activity for users on the campus network.  A better name for this post might be, “What does local use of ResearchGate look like?”

Consider this problem: Two fixed-length text files, one has 42,000 lines while the other has 13,000.  In each file, a single line represents information about a particular user. If a person with status ‘X’ in the first file also appears in the 2nd file with a status of ‘Y’, we need to keep the line in the first file and delete the line for that user from the 2nd file. We can match up the user between files via the person’s ID# field which appears in both files.

Real word example: We have two different fixed-length files that we receive weekly from the campus computer center. For years we just sent those files (in Voyager SIF format) directly into our Voyager system to update patrons (students with one file, faculty and staff with the other). Moving to Alma we decided the best course for our accelerated implementation was to let the computer center continue producing those SIF files and we’d take on the task of converting the information into the XML form that Alma was expecting. That did require a bit of code but it’s been working pretty well.  But not perfectly…

Some people appear in both files and the last file loaded sets the borrower status for that individual.  For example, the grad student who also works in a department somewhere on campus ends up having the same borrowing privileges as a wages worker in the food court once we overwrite the student status from Student Information System (SIS) file with the same user’s record in the file we get from HR that shows a status of “wages.”

Understandably, a student complained about this the other day and ever since I’ve been trying to think of a programmatic fix.  But before walking through my solution—and to make what I came up with a bit easier to follow–here’s a sample line from each of the two files

From the SIS file (42,000 lines)

00000 ro medm 1 2019.06.102020  03.2 54 G00009705 Jones Seneca T. 2205 Any Street, Anytown, Virginia [line continues…]

From the HR file (13,000 lines)

00000 po ireg 1 2019.06.102020 03.03 2803 G00223705  Everyman, Bob E. 1422 Stafford Street, Apt 7, Arlington, VA 22203 [line continues…]

There must be many ways to solve this problem. My first thought was a Perl program but my brute force algorithm (look for a grad student in the first file, then scan the 2nd file for a match) seemed too slow (stupid?) to pursue.  Also considered OpenRefine but that quickly got very fiddly and I was looking for a server-side solution I could add into the script that handles the conversion to XML prior to sending the data to Alma.  Finally, I thought, “what about a shell script where awk and grep do the heavy lifting?”

My algorithm ended up looking something like this:

1. Examine each line of the Student Information System file and where we find “medm” on a line in the 3rd column (a code for grad students), extract the Mason ID number (in column 8), and write just that ID# to the output file.

2. Examine each line of the HR file and write an output file that contains just the Mason ID number for anyone with the status of “wages” (matching records have “ireg” in the 3rd column).

3. Go through both files and where the Mason ID number in file one matches a Mason ID number in the 2nd file, add that ID# to a third file (contains only ID numbers that are in both original files).

4. Finally, grep the two files using an inverted match (e.g., -vf ) writing out lines from the original HR file that are not in the file we created in step 3 (where ID appeared in both original files).  This is equivalent to deleting lines from file the HR.dat file for the wages person who appeared in SIS file as a grad student. Voilà! The grad status record doesn’t get overwritten when the processed HR file (hr_final.out) is loaded into Alma.

Four-step solution looks like this:

awk '{ if ($3 =="medm") { print $ 8}}' < SIS.dat > grad_IDs.txt

awk '{ if ($3 =="ireg") { print $ 8}}' <  HR.dat > wages_IDs.txt

awk -F\| 'NR==FNR{a[$1]++;next}a[$1]' grad_IDs.txt wages_IDs.txt > both.txt

grep -vf both.txt HR.dat > hr_final.out

Note: That third line has a pipe character after the -F\, not a capital “I” … and yes, for that third line I stand on the shoulders of giants who post answers on Stack Overflow

How long does it take this to run on a rather old RedHat 6.5 server?  1.5 seconds

Here’s a nice awk cheatsheet:

https://www.cheatography.com/tme520/cheat-sheets/awk-english/pdf/

The basic idea is this:  each line of our proxy server’s log file contains the user’s netID, the time of the transaction and the link that was proxied.   Another dataset I have access to–the student directory—includes both the netID and the declared major for each student.  Joining these two datasets on the netID, I can get a count of e-content activity for each major.

This works but obviously it’s not a very precise tool.  For example, I’m only analyzing traffic that goes through our proxy server (basically our off-campus users) and ignoring open-access e-content altogether (we don’t use our proxy server to provide access to those sorts of resources).     Six thousand of our 34,000 students live on campus so I’m comfortable assuming that off-campus use is a reasonable proxy for all usage but I realize this is a very blunt instrument.

To explain the graph below, that 986,024 figure for Psychology majors means in the semester-long  activity log file on the proxy server, there were 986,024 lines in the file that were generated by content coming through the server bound for a declared Psychology major.

The method I use to convert the proxy server log to an SQL database hasn’t changed much from the how-to I posted a few years ago.  I did, however, make a small, simple and obvious refinement this year: before computing the per-major count, I eliminated lines from the log file that referenced things like javascript or css files and also tossed out glyph or icon entries.  Doing so removed a bit of noise from my results since some databases have much more graphically-rich interfaces and all those widgets, icons and glyphs skew my count.  Here’s a graphic representation of the top 50 or so e-content using majors:

This next chart shows the number of students in each major.    Comparing this graph with the first one, you can see that some majors generate far higher levels of use than their numbers might indicate (e.g., Public Policy ranks 40th in the number of students majors but 15th in e-content usage).  

That changed one afternoon this summer as I grabbed my laptop and tagged along with them to a meeting with the Associate Provost for Undergraduate Education to talk about OERs.

As the meeting progressed (and moved ever further from my area of expertise) I started stealing moments to jump in and out of various OER aggregation sites, curious to see the sorts of resources already available on the net.

If you’ve spent much time with OERs, you won’t be surprised to hear that I discovered:

• many dissimilar aggregations of content;
• so many wildly-different interfaces;
• so much duplication across these aggregations;
• and such inconsistent metadata.

As I poked around, I could easily envision a faculty member—excited by idea of OERs—feeling the enthusiasm drain away as she dove in and out of the various content silos.   Soon I found myself thinking much less about OERs and far more about how to improve their discoverability as a way to improve OER adoption…

Already familiar with Deep Web Technologies Explorit software (we use it for a number of local search engines like our popular Sciences Metafinder), I realized that discovery of OER materials presents a problem that’s tailor-made for a federated search solution.   Looking across OER sources we find:

• a large number of search targets (a federated search would save hundreds of clicks),
• and fortunately each site is more-or-less focused (that’s good, minimizes noise in retrieval sets).
• We see redundant content across many of these sources (de-duping retrieval would be a huge win, too bad eccentric metadata makes that difficult), and
• there’s a vertigo-inducing variety of search interfaces (distilling that to one would be great, wouldn’t it?).

Beyond improving the discovery process, I realized building our own search engine would also give us the opportunity to take a more expansive view of what constitutes an OER–by searching the more common OER repositories but also hitting sites that offer quality, open educational content even if that isn’t their sole or even primary purpose. Sites like DPLA, HathiTrust (of particular value where the educator belongs to a HathiTrust member institution), Internet Archive, and World Digital Library to name a few.

I pitched the idea to Deep Web Technologies. Abe Lederman, CEO was very enthusiastic and offered to help us turn the idea around quickly.  True to his word, within just a few weeks we had a powerful OER discovery service ready to go.

The Mason OER Metafinder launches…

I tweeted once or twice about the Metafinder’s existence and shared the link with a few people…but outside Mason we weren’t seeing much interest or activity.  Some weeks later, Deep Web posted a story about the OER Metafinder on their tech blog.   That was picked up by Information Today and other corporate PR news sites and the Metafinder began to build a small audience but still nothing dramatic. I didn’t feel we had failed exactly–the metafinder was impressing local audiences–but I was certainly disappointed. We couldn’t seem to get what I considered a great tool more widely used.

Three or four weeks passed then one morning I saw a message from Abe Lederman in my inbox, subject “OER Metafinder going viral?”   Abe said they had seen a spike, hundreds of hits, on the Metafinder in the past two hours–mostly from the Northeast US.  He asked, “Any idea why?”

I didn’t have a clue but later that afternoon, I got an email mentioning this post on the SPARC Libraries and OER Forum earlier in the day:

I don’t remember seeing this announcement on any of our OER lists last month, or at OpenEd, but one of our library liaisons just forwarded it to me. It’s the announcement of an aggregated OER search engine created by George Mason and a web tech company, which looks, on the face of it, to be a “Google for OER”. It searches many open archival/book repositories (DPLA, HathiTrust, Internet Archive) as well as the standard OER ones (Merlot, OTL, OpenStax, etc.) and has some great limiters to narrow down results. Congrats GMU!

This unsolicited mention on a listserv aimed at precisely the right group of people proved catalytic.  Within two days, I searched the LibGuides community site and found 28 institutions already linking to our OER Metafinder.   Noticing that it was catching on with LibGuides users, I added sample search widget code to our “About the Metafinder” page.   Searching https://community.libguides.com and crawling the web with DEVONagent Pro (a great personal research app for Mac, by the way), I see that just a few weeks after Abe’s initial “gone viral” email there are already more than 100 sites linking to our Metafinder.

7/2019 update:  We now have 348+ sites linking to MOM

Click the image below to take a peek at our ever-evolving sandbox:

Of course, that doesn’t mean there aren’t a host of smaller services we can build around social media. By way of example, here are three social media services we offer the Mason community. One’s pretty simple while the other two require a bit more infrastructure.

Mason Tweets (http://tweet.gmu.edu)

This curated feed from “official” and “near-official” twitter accounts from across the university offers a quick and easy way to take the “Mason Nation” pulse.

To produce this service, we created a MasonTweeter account on Twitter to follow Mason-related feeds.  The web presence is simply a page that embeds the MasonTweeter timeline.

Preztweets (http://preztweets.gmu.edu)

An archive of every tweet from Mason’s President, Ángel Cabrera.

This service stems from a discussion I had with Dr. Cabrera a few years ago.  At that time, Twitter did not offer users an archive of their tweets (they do now), so we were looking into how we might save his tweets for future university historians.  We settled on a method that offers a searchable database of tweets stored locally in a MySQL database (suitable for future archiving).  Thanks to Andrew M. Whalen for the code that helped build this LAMP-based archiving service.

Social Feed Manager (SFM) (https://gwu-libraries.github.io/sfm-ui/

Just the other day, I set up our most ambitious social media service yet: Social Feed Manager.

SFM is a Django application developed by George Washington University Libraries to collect social media data from Twitter. It connects to Twitter’s approved API to collect data in bulk and makes it possible for scholars, students, and librarians to identify, select, collect, and preserve Twitter data for research purposes. We’re running SFM in a Docker container (using Docker for Mac) which simplifies installation and abstracts away much of the underlying complexity.

We have added Social Feed Manager to the suite of data services we offer out of the new Digital Scholarship Center we’ve been shaking down in beta since late January.