Collaborative Manuscript Transcription

Typologie des méthodes de contrôle de la qualité dans les projets de crowdsourcing

noreply@blogger.com (Ben W. Brumfield) — Tue, 14 May 2013 14:29:00 +0000

A translation of my 2012-03-05 post "Quality Control for Crowdsourced Transcription" which appeared in "Etat de l’art en matière de Crowdsourcing dans les bibliothèques numériques" by Moirez, Moreaux, and Josse (2013), reproduced for Francophone readers:

«Single-track methods»: le document ne fait l’objet que d’une seule transcription (par un seul contributeur ou de façon collaborative ensemble sur le même document)
1. «Open-ended community revison»: (Wikipédia) les utilisateurs peuvent continuer à modifier le texte transcrit, sans limite dans le temps. Un historique des modifications permet de revenir à la version précédente et d’éviter le vandalisme.
2. «Fixed-term community revision» (Transcribe Bentham) : convient pour des projets d’édition plus traditionnels, dont l’objectif est la publication d’une “version finale”. Quand une transcription atteint un niveau acceptable, val idée par les experts, elle est close et publiée.
3. «Community-controlled revision workflows» (Wikisource) : la transcription est considérée comme une “version finale” non plus par des experts, mais parce qu’elle a traversé un workflow collaboratif de correction/révision/validation -
4. «Transcriptions with "known-bad" insertions before proofreading» : dans une première phase, les correcteurs sont invités à transcrire. Puis d’autres correcteurs révisent la transcription en la comparant au texte original; pour s’assurer que la seconde lecture est bien réalisée, des erreurs sont ajoutées dans le texte: si toutes les «fausses erreurs» sont corrigées, le système déduit que les «vraies erreurs» ont dû être corrigées aussi.
5. «Single-keying with expert review» : lorsqu’une transcription a été réalisée par un contributeur, elle est validée ou rejetée par un expert (soit un professionnel de l’institution à l’origine du projet, soit un contributeur sélectionné). Si la correction est rejetée, elle est soit à nouveau soumise à correction, soit corrigée par l’expert et validée.
«Multi-track methods»: ces méthodes conviennent particulièrement à des corrections portant sur des données structurées ou des micro-tâches. La même image de départ est présentée à plusieurs contributeurs qui transcrivent chacun à partir de zéro. Généralement, les contributeurs ne savent pas s’ils sont les premiers correcteurs ou si d’autres transcriptions ont déjà été soumises. Puis les données ainsi collectées sont comparées automatiquement.
1. «Triple-keying with voting» (Old Weather, ReCAPTCHA) : l’image est présentée à 3 contributeurs, la majorité l’emporte (au depart, Old Weather proposait l’image à 10 contributeurs, mais ils se sont aperçus que la pertinence était sensiblement la même avec 3 qu’avec 10 contributeurs)
2. «Double-keying with expert reconciliation»: la même donnée est présentée à deux contributeurs, et, s’ils ne sont pas d’accord entre eux, un expert tranche.
3. «Double-keying with emergent community-expert reconciliation» (FamilySearch Indexing): la method est presque similaire à la précédente, sauf que l’expert qui tranche entre deux corrections divergentes est lui-même un contributeur, qui a été promu conciliateur grâce à l’analyse automatique de ses contributions (volume,pertinence).
4. «Double-keying with N-keyed run-off votes»: si les deux contributeurs ne sont pas d’accord, la correction est re-proposée à un nouveau duo/trio d’usagers.

Itinera Nova in the World(s) of Crowdsourcing and TEI

noreply@blogger.com (Ben W. Brumfield) — Mon, 29 Apr 2013 18:23:00 +0000

On April 25, 2013, I presented this talk at the International Colloquium Itinera Nova in Leuven, Belgium. It was a fantastic experience, which I plan to post (and speak) more about, but I wanted to get my slides and transcript online as soon as possible.

Abstract: Crowdsourcing for cultural heritage material has become increasingly popular over the last decade, but manuscript transcription has become the most actively studied and widely discussed crowdsourcing activity over the last four years. However, of the thirty collaborative transcription tools which have been developed since 2005, only a handful attempt to support the Text Encoding Initiative (TEI) standard first published in 1990. What accounts for the reluctance to adopt editorial best practices, and what is the way forward for crowdsourced transcription and community edition? This talk will draw on interviews with the organizers behind Transcribe Bentham, MoM-CA, the Papyrological Editor, and T-PEN as well as the speaker's own experience working with transcription projects to situate Itinera Nova within the world of crowdsourced transcription and suggest that Itinera Nova's approach to mark-up may represent a pragmatic future for public editions.

I'd like to talk about Itinera Nova within the world of crowdsourced transcription tools, which means that I need to talk a little bit about crowdsourced transcription tools themselves, and their history, and the new things that Itinera Nova brings.

Crowdsourced transcription has actually been around for a long time. Starting in the 1990s we see a number of what are called "offline" projects. This is before the term crowdsourcing was invented.

A Dutch initiative: Van Papier naar Digitaal which is transcribing primarily genealogy records.
FreeBMD, FreeREG, and FreeCEN in the UK, transcribing church registers and census records.
Demogen in Belgium -- I don't know a lot about this -- it appears to be dead right now, but if anyone can tell me more about this, I'd like to talk after this.
Archivalier Online--also transcribing census records--in Denmark,
And a series of projects by the Western Michigan Genealogy Society to transcribe local census records and also to create indexes of obituaries.

One thing these have in common, you'll notice, is that these are all genealogists. They are primarily interested in person names and dates. And they emerge out of an (at least) one hundred year old tradition of creating print indexes to manuscript sources which were then published. Once the web came online, the idea of publishing these on the web [instead] became obvious. But the tools that were used to create these were spreadsheets that people would use on their home computers. Then they would put CD ROMs or floppy disks in the posts and send them off to be pubished online.

Really the modern era of crowdsourced transcription begins about eight years ago. There are a number of projects that begin development in 2005. They are released (even though they've been in development for a while) starting around 2006. Familysearch Indexing is, again, a genealogy system primarily concerned with records of genealogical interest which are tabular. It is put up by the Mormon Church.

Then things start to change a little bit. In 2008, I publish FromThePage, which is not designed for genealogy records per se -- rather it's designed for 19th and 20th century diaries and letters. (So here we have more complex textual documents.) Also in 2008, Wikisource--which had been a development of Wikipedia to put primary sources online--start using a transcription tool. But initially, they're not using it for manuscripts because of policy in the English, French, and Spanish language Wikisources. The only people using it for manuscripts are the German Wikisource community, which has always been slightly separate. So they start transcribing free-form textual material like war journals [ed: memoirs] and letters. But again, we have a departure from the genealogy world.

In 2009, the North American Bird Phenology Program starts transcribing bird observations. So in the 1880s you had amateur bird-watchers who would go into the field and they would record their sightings of certain ducks, or geese, or things like that, and they would record the location and the birds they had observed. So we have this huge database of the presences of species throughout North America that is all on index cards. And as the climate changes and habitats change, those species are no longer there. So scientists who want to study bird migration and climate change need access to these. But they're hand-written on 250,000 index cards, so they need to be transformed. So that requires transcription, also by volunteers. [ed: The correct number of cards is over 6 million, according to Jessica Zelt's "Phenology Program (BPP): Reviving a Historic Program in the Digital Era"]

2010 is the year that crowdsourced transcription really gets big. The first big development is the Old Weather project, which comes out of the Citizen Science Alliance and the Zooniverse team that got started with GalaxyZoo. The problem with studying climate change isn't knowing what the climate is like now. It is very easy to point a weather satellite at the South Pacific right now. The problem is that you can't point a weather satellite at the South Pacific in 1911. Fortunately, in many of the world's navies, the officer of the watch would, every four hours, record the barometric pressure, the temperature, the wind speed and direction, the latitude and the longitude in the ships logs. So all we have to do is type up every weather observation for all the navies' ships, and suddenly we know what the climate was like. Well, they've actually succeeded at this point -- in 2012 they finished transcribing all the British Royal Navy's ships log weather observations during World War I. So this has been very successful -- it's a monumental effort: they have over six hundred thousand registered accounts--not all of those are active, but they have a very large number of volunteers.

Also in 2010 in the UK, Transcribe Bentham goes live. (We'll talk a lot more about this -- it's a very well documented project.) This is a project to transcribe the notes and papers of the utilitarian philosopher Jeremy Bentham. It's very interesting technically, but it was also very successful drawing attention to the world of crowdsourced transcription.

In 2011, the Center for History and New Media at George Mason University in northern Virginia published the Papers of the United States War Department, and builds a tool called Scripto that plugs into it. Now this is primarily of interest to military and social historians, but again we're getting away from the world of genealogy, we're getting away from the world of individual tabular records, and we're getting into dealing with documents.

Once we get there, we have a tension. And this is a pretty common tension. There's an institutional tension, in that editing of documents has historically been done by professionals, and amateur editions have very bad reputations. Well now we're asking volunteers to transcribe. And there's a big tension between, well how do volunteers deal with this [process], do we trust volunteers? Wouldn't it be better just to give us more money to hire more professionals? So there's a tension there.

There's another tension that I want to get into here, since today is the technical track, and that's the difference between easy tools and powerful tools, and [the question of] making powerful tools easy to use. This is common to all technology--not just software, and certainly not just crowdsourced transcription--but it's new because this is the first time we're asking people to do these sorts of transcription projects.

Historically these professional [projects] have been done using mark-up to indicate deletions or abbreviations or things like that.

So there's this fear: what happens when you take amateurs and add mark-up?

Well, what is going to happen? Well, one solution--and it's a solution that I'm distressed to say is becoming more and more popular in the United States--is to get rid of the mark-up, and to say, well, let's just ask them to type plain text.

There's a problem with this. Which is that giving users power to represent what they see--to do the tasks that we're asking them to do--enables them. Lack of power frustrates them. And when you're asking people to transcribe documents that are even remotely complex, mark-up is power.

So I'm going to tell a little story about scrambled eggs. These are not the scrambled eggs that I ate this morning--which were delicious by the way--but they're very similar.

I'm going to pick on my friends at the New York Public Library, who in 2011 launched the "What's on the Menu?" project. They have an enormous collection of menus from around the world, and they want to track to culinary history of the world as dishes originate in one spot and move to other locations, the change in dishes--when did anchovies become popular? Why are they no longer popular?--things like that. So they're asking users to transcribe all of these menu items. They developed a very elegant and simple UI. This UI did not involve mark-up; this is plain-text. In fact--I'm going to get over here and read this--if you look at this instruction, this is almost stripped text: "Please type the text of the indicated dish exactly as it appears. Don't worry about accents."

Well, this may not be a problem for Americans, but it turns out that some of their menus are in languages that contain things that American developers might consider accents. This is a menu that was published on their site in 2011. They sent out an appeal asking, "can anyone read Sütterlin or old German Kurrentschrift"? I saw this and I went over to a chat channel for people who are discussing German and the German language, because I knew that there were some people familiar with German paleography there, and I wanted to try it out.

So the transcribers are going through and they're transcribing things, and they get to this entry: Rühreier. All right, let's transcribe that without accents. So they type in what they see. Rühreier is scrambled eggs. And what they type is converted to "Ruhreier", which are... eggs from the Ruhrgebiet? I don't know? This is not a dish. I'm not familiar with German cuisine, but I don't think that the Ruhr valley is famous for its eggs.

And this is incredibly frustrating! We see in the chat room logs: "Man, I can't get rid of 'Ruhreier' and this (all-capital) 'OMELETTE'! What's going on? Is someone adding these back? Can you try to change "Ruhreier" to "Rühreier"? It keeps going back!"

So we have this frustration. We have this potential to lose users when we abandon mark-up; when we don't give them the tools to do the job that we're asking them to do.

Okay. Let's shift gears and talk about a different world. This is the world of TEI, the Text Encoding Initiative. It's regarded as the ultimate in mark-up -- Manfred [Thaller] mentioned it some time earlier. It's been a standard since 1990, and it's ubiquitous in the world of scholarly editing.

Remember, up until recently, all scholarly editing was done by professionals. These professionals were using offline tools to edit this XML which Manfred described as a "labyrinth of angle brackets." It was never really designed to be hand-edited, but that's what we're doing.

And because it's ubiquitous and because it's old, there's a perception among at least some scholars, some editors, that this is just a 'boring old standard'. I have a colleague who did a set of interviews with scholars about evaluating digital scholarship, and not all but some of the responses she got when she brought up TEI were "TEI? Oh, that's just for data entry."

Well, not quite. TEI has some strengths. It is an incredibly powerful data model. The people who are doing this--these professionals who have been working with manuscripts for decades--they've developed very sophisticated ways of modeling additions to texts, deletions to texts, personal names, foreign terms -- all sorts of ways of marking this up.

It has great tools for presentation and analysis. Notice I didn't say transcription.

And it has a very active community, and that community is doing some really exciting things.

I want to use just one example of something that has only been around in the last four years that it's been developed. It's a module that was created for TEI called the Genetic Edition module. A "genetic edition" is the idea of studying a text as it changes -- studying the changes that an author has made as they cross through sections and created new sections, or over-written pieces.

So it's very sophisticated, and I want to show you the sorts of things you can do [with it] by demostrating an example of one of these presentation tools by Elena Pierazzo and Julie Andre. Elena's at King's College London, and they developed this last year.

This is a draft of--I believe it's Proust's Recherches du Temps Perdu--unfortunately I can't see up there. But as you can see, this is a very complicated document. The author has struck through sections and over-written them. He's indicated parts moved. He's even -- if you look over here -- he's pasted on an extra page to the bottom of this document. So if you can transcribe this to indicate those changes, then you can visualize them.

[Demo screenshots from the Proust Prototype.] And as you slide, you see transcripts appear on the page in the order that they're created,

And in the order that they're deleted even.

There's even rotation and stuff --

It's just a brilliant visualization!

So this is the kind of thing that you can do with this powerful data model.

But how was that encoded? How did you get there?

Well, in this case, this is an extension to that thousand-page book. It's only about fifty pages long, printed, and it contains individual sets of guidelines. In this case, this is how Henrik Ibsen clarified a letter. In order to encode this, you use this rewrite tag with a cause... And this is that forest of angle brackets; this is very hard. And this is only one item from this document of instructions, which was small enough that I could cut it out and fit it on a slide.

So this is incredibly complex. So if TEI is powerful; and if, as it gets more complex, it becomes harder to hand-encode; and as we start inviting members of the public and amateurs to participate in this work, how are we going to resolve this?

If there's a fear about combining amateurs and mark-up, what do we do when we combine amateurs with TEI? This is panic!

And it is very rarely attempted. I maintain a directory of crowdsourced transcription tools, with multiple projects per tool. And of the 29 projects in this directory, only 7 claim to support TEI.

One of them is Itinera Nova. I found out about this when I was preparing a presentation for the TEI conference last year, in which I interviewed people running projects doing this crowdsourcing, and found out about their experience of users trying to encode in TEI, and asked, "Do you know anyone else?"

And that's how I found out about Itinera Nova, which is unfortunately not very well known outside of Belgium. This is something that I hope to part of correcting, because you have a hidden gem here -- you really do. It is amazing.

So how do you support TEI? Well, one approach--the most common approach--is to say we'll have our users enter TEI, but we'll give them help. We'll create buttons that add tags, or menus that add tags. This has been the approach taken by T-PEN (created by the Center for Digital Thelogy out of Saint Louis University), and a project associated with them, the Carolingian Canon Law Project. It's also the approach taken by Transcribe Bentham with their TEI toolbar. Menus are an alternative, but essentially the do the same thing -- they're a way of keeping users from typing angle brackets. So the Virtuelles deutsches Urkundennetzwerk is one of those, as well as the Papyrological Editor which is used by scholars studying Greek papyri.

So how well does that work? You provide users with buttons that add tags to their text. Here's an example from Transcribe Bentham.

Here's an example from Monasterium. And the results are still very complicated. The presentation here is hard. It's hard to read; it's hard to work with.

That does not mean that amateurs cannot do it at all! Certainly the experience of Transcribe Bentham proves that amateurs to the same level as any professional transcriber, using these tools and coding these manuscripts, even without the background.

But there are limitations. One limitation is that users outgrow buttons. In Transcribe Bentham, [the most active] users eventually just started typing the angle brackets themselves -- they returned to that labyrinth of angle brackets of TEI tags.

Another problem is more interesting to me, which is when users ignore buttons. Here we have one editor who's dealing with German charters, who uses these double-pipes instead of the line break tag, because this is what he was used to from print. This speaks to something very interesting, which is that we have users who are used to their own formats, they're used to their own languages for mark-up, they're used to their own notations from print editions that they have either read or created themselves. And by asking them to switch over to this style of tagging, we're asking them not just to learn something new, but also to abandon what they may already know.

And, frankly, it's really hard to figure out which buttons [to support]. Abigail Firey of the Carolingian Canon Law Project talks about how when they were designing their interface, they had 67 buttons. This is very hard to navigate, and the users would just give up and start typing angle brackets instead, because buttons aren't a magic solution.

This is where Itinera Nova comes in. The "intermediate notation" that Professor Thaller was talking about is quite clear-cut, and it maps well to the print notations that volunteers are already used to.

And what's interesting about this is that what many people may not realize is that Itinera Nova--despite having a very clear, non-TEI interface--has full TEI under the hood.

Everything is persisted in this TEI database, so the kinds of complex analysis that we talked about earlier--not necessarily the Proust genetic editions, but this kind of thing--is possible with the data that's being created. It's not idiosyncratic.

So as a result, I really think that in this, Itinera Nova points the way to the future. Which is to abandon this idea that TEI is just for data entry, or that amateurs cannot do mark-up. Both of those ideas are bogus! Instead, let's say: use TEI for the data model; for the presentation, so we have these beautiful sliders. And whatever else will get created out of the annotation tool, out of the transcription tool, let's use that for the data model and for the presentation. But let's consider let's consider hooking up these--I don't want to say "easier"--but these more straightforward, these more traditional user interfaces [for transcription].

This is something that I think is really the way forward for crowdsourced transcription. It is being done right now by the Papyrological Editor, it has been done by Itinera Nova for a long time. And there are now some incipient projects to move forward with this. One of these is a new project at the University of Maryland, Maryland Institute for Technology and the Humanities, the Skylark project, in which they are taking those same transcription tools that were used for Old Weather to allow people to mark up and transcribe portions of an image of a literary text that has been heavily annotated--like that Proust--to create data using the data model that can be viewed with tools like the Proust viewer.

So this is, I think, the technical contribution that Itinera Nova is making. Obviously there are a lot more contributions--I mean I'm absolutely stunned by the interaction with the volunteer community that's happening here--but I'm staying on the technical track, so I'm not going to get into that.

Are there any questions? No? Keep up the great work -- you folks are amazing.

Ngoni Munyaradzi on Transcribe Bleek and Lloyd

noreply@blogger.com (Ben W. Brumfield) — Tue, 26 Feb 2013 20:59:00 +0000

Ngoni Munyaradzi is a Master's student in Computer Science at the University of Cape Town, South Africa, working on a research project on the transcription of the Digital Bleek and Lloyd collection. He kindly agreed to an interview over email, which I present below:

Your website does an excellent job explaining the background and motivation of Transcribe Bleek and Lloyd. Can you tell us more about the field notebooks you are transcribing?

The Digital Bleek and Lloyd Collection is composed of dictionaries, artwork and notebooks documenting stories about the earliest inhabitants of Southern Africa, the Bushman people. The notebooks were written by Wilhelm Bleek, his sister-in-law, Lucy Lloyd and Dorothea Bleek (Wilhelm's daughter) in the 19th century, with the help of a number of Bushmen people who were prisoners in the Western Cape region of South Africa at the time. The notebooks were recorded in the |Xam and !Kun languages and English translations of these languages are available in the notebooks.

Link to the collection: http://lloydbleekcollection.cs.uct.ac.za/

Correct me if I'm wrong, but it seems like at least in the case of |Xam, you are working with one of the only representatives of an extinct language. Are there any standard data models for these kinds of vocabularies/bilingual texts which you're using?

There are no complete models - the best known models are still only partial.

I suspect that I'm not alone in wondering why these Bushman people were prisoners during the writing of these texts. Can you tell us a bit more about the Bleek/Lloyd informants, or point us to resources on the subject?

The bushman people were prisoners because of petty crimes and a grossly unfair colonial government. On the Bleek and Lloyd website there is a story on each contributor. There is information in various books on the subject as well, but I am not sure there is more that is known than what is on the website. see:
http://lloydbleekcollection.cs.uct.ac.za/xam.html
http://lloydbleekcollection.cs.uct.ac.za/kun.html

This is the first transcription project I'm aware of using the Bossa Crowd Create platform. What are the factors that led you to choose that platform and what's been your experience setting it up?

In 2011 when our project began Bossa was the most mature opensource crowdsourcing framework that was tailored for volunteer projects available. Due to this Bossa suited well with the project's requirements. The alternative crowdsourcing frameworks available at the time used payment methods.

Setting up the Bossa framework was a relatively straight-forward task. The documentation online is very thorough and with examples of how to set-up test applications. I also got assistance from David Anderson the developer of Bossa.

The Bushman writing system seems extremely complex with it's special characters and multiple diacritics. I see that you are using LaTeX macros to encode these complexities. Why did you decide on LaTeX and what has been the user response to using that notation?

So the project is part of ongoing research related to the Bleek and Lloyd Collection within our Digital Libraries Laboratory at the University of Cape Town. Credit for developing the encoding tool goes to Kyle Williams. And the reason why he chose to use LaTeX was that; using custom LaTeX macros allowed for both the problem of the encoding and visual rendering of the text to be solved in a single step. Developing a unique font for the Bushman script is something we might look at in the future!

Here's a link to a paper published on the encoding tool developed by Kyle Williams: http://link.springer.com/chapter/10.1007%2F978-3-642-24826-9_28

Overall the user feedback has been good, as most users are able to complete transcriptions using the LaTeX macros. We have gotten suggestions from users to use glyphs to encode the complexities. Currently the scope of my masters research project does not include that. There are talks in our research group to develop a unique font to represent the |Xam and !Kun languages, as this is not supported by Unicode.

User 1 Comment: "I think the palette handles the complexity of the character set very well. This material is inherently difficult to transcribe. The tool has, on the whole, been well thought out to meet this challenge. I think it needs to be improved in some ways, but considering the difficulties it is remarkably well done."

User 2 Comment: "VERY intuitive, after a few practice transcriptions. I actually enjoyed using the tool after a page was done."

This is incredibly useful. So far as I'm aware, yours is only the third crowdsourced transcription project that's surveyed users seriously (after the North American Bird Phenology Project and Transcribe Bentham). Do you have any advice on collecting user feedback at such an early stage?

Collecting user feedback in the early stages will tremendously help project administrators determine whether the setup of the project is easy to follow for participants. One can easily pick up any hindrances to user participation and address these early. From our project, I've found that participants can actually suggest very helpful ideas that will make the data collection process better.

Crowdsourced citizen science and cultural heritage projects have mostly been based in the USA, Northern Europe and Australia until recently -- in fact, yours is the first that I'm aware of originating in sub-Saharan Africa. I'd really like to know which projects inspired your work with Transcribe Bushman, and what your hopes are for crowdsourced transcription projects focusing on Africa?

Our work was mostly inspired by the success of GalaxyZoo at recruiting volunteers, and also the Transcribe Bentham project that explored the feasibility of volunteers performing transcription. I hope that more crowdsourced transcription projects will start-up within Africa in the near future. What would be interesting is to see a transcription project for the Timbuktu manuscripts of Mali. Beyond transcription, I would like to see other researchers adopting crowdsourcing in fields of specialty within Africa.

Thanks so much for this interview. If people want to help out on the project, what's the best way for them to contribute?

Interested participants can simply:

Create an account on the project website.
Watch a 5 minute video tutorial on how to transcribe the Bushman languages.
With that, you are ready to start transcribing pages.

Detecting Handwriting in OCR Text

noreply@blogger.com (Ben W. Brumfield) — Mon, 25 Feb 2013 14:56:00 +0000

This is my fourth and final post about the iDigBio Augmenting OCR Hackathon. Prior posts covered the hackathon itself, my presentation on preliminary results, and my results improving the OCR on entomology specimens. The other participants are slowly adding their results to the hackathon wiki, which I recommend checking back with (their efforts were much more impressive than mine).

Clearly handwritten: T=8, N=78% from terse and noisy OCR files

Let's say you have scanned a large number of cards and want to convert them from pixels into data. The cards--which may be bibliography cards, crime reports, or (in this case) labels for lichen specimens--have these important attributes:

They contain structured data (e.g. title of book, author, call number, etc. for bibliographies) you want to extract, and
They were part of a living database built over decades, so some cards are printed, some typewritten, some handwritten, and some with a mix of handwriting and type.

The structured aspect of the data makes it quite easy to build a web form that asks humans to transcribe what they see on the card images. It also allows for sophisticated techniques for parsing and cleaning OCR (which was the point of the hackathon). The actual keying-in of the images is time consuming and expensive, however, so you don't want to waste human effort on cards which could be processed via OCR.

Since OCR doesn't work on handwriting, how do you know which images to route to the humans and which to process algorithmically? It's simple: any images that contain handwriting should go to the humans. Detecting the handwriting on the images is unfortunately not so simple.

I adopted a quick-and-dirty approach for the hackathon: if OCR of handwriting produces gibberish, why send all the images through a simple pass of OCR and look in the resulting text files for representative gibberish? In my preliminary work, I pulled 1% of our sample dataset (all cards ending with "11") and classified them three ways:

Visual inspection of the text files produced by an ABBY OCR engine,
Visual inspection of the text files produced by the Tesseract OCR engine, and
Looking at the actual images themselves.

To my surprise, I was only able to correctly classify cards from OCR output 80% of the time -- a disappointing finding, since any program I produced to identify handwriting from OCR output could only be less accurate. More interesting was the difference between the kinds of files that ABBY and Tesseract produced. Tesseract produced a lot more gibberish in general--including on card images that were entirely printed. ABBY, on the other hand, scrubbed a lot of gibberish out of its results, including that which might be produced when it encountered handwriting.

This suggested an approach: look at both the "terse" results from ABBY and the "noisy" results from Tesseract to see if I could improve my classification rate.

Easily classified as type-only, despite (non-characteristic) gibberish: T=0,N=0 from terse and noisy OCR files.

But what does it mean to "look" at a file? I wrote a program to loop through each line of an OCR file and check for the kind of gibberish characteristic of OCR and handwriting. Inspecting the files reveals some common gibberish patterns, which we can sum up as regular expressions:

GARBAGE_REGEXEN = {
  'Four Dots' => /\.\.\.\./,
  'Five Non-Alphanumerics' => /\W\W\W\W\W/,
  'Isolated Euro Sign' => /\S€\D/,
  'Double "Low-Nine" Quotes' => /„/,
  'Anomalous Pound Sign' => /£\D/,
  'Caret' => /\^/,
  'Guillemets' => /[«»]/,
  'Double Slashes and Pipes' => /(\\\/)|(\/\\)|([\/\\]\||\|[\/\\])/,
  'Bizarre Capitalization' => /([A-Z][A-Z][a-z][a-z])|([a-z][a-z][A-Z][A-Z])|([A-LN-Z][a-z][A-Z])/,
  'Mixed Alphanumerics' => /(\w[^\s\w\.\-]\w).*(\w[^\s\w]\w)/
}

However, some of these expressions match non-handwriting features like geographic coordinates or bar codes. Handling these requires a white list of regular expressions for gibberish we know not to be handwriting:

WHITELIST_REGEXEN = {
  'Four Caps' => /[A-Z]{4,}/,
  'Date' => /Date/,
  'Likely year' => /1[98]\d\d|2[01]\d\d/,
  'N.S.F.' => /N\.S\.F\.|Fund/,
  'Lat Lon' => /Lat|Lon/,
  'Old style Coordinates' => /\d\d°\s?\d\d['’]\s?[NW]/,
  'Old style Minutes' => /\d\d['’]\s?[NW]/,
  'Decimal Coordinates' => /\d\d°\s?[NW]/,  
  'Distances' => /\d?\d(\.\d+)?\s?[mkf]/,  
  'Caret within heading' => /[NEWS]\^s/,
  'Likely Barcode' => /[l1\|]{5,}/,
  'Blank Line' => /^\s+$/,
  'Guillemets as bad E' => /d«t|pav«aont/  
}

With these on hand, we can calculate a score for each file based on the number of occurrences of gibberish we find per line. That score can then be compared against a threshold to determine whether a file contains handwriting. Due to the noisiness of the Tesseract files, I found it most useful to calculate their score N as a percentage of non-blank lines, while the score for the terse files T worked best as a simple count of gibberish matches.

Threshold	Correct	False Positives	False Negatives
T > 1 and N > 20%	82%	10 of 45	8 of 60
T > 0 and N > 20%	84%	13 of 45	4 of 60
T > 1	79%	10 of 45	12 of 60
N > 20%	75%	8 of 45	18 of 60
N > 10%	81%	14 of 45	6 of 60

One interesting thing about this approach is that adjusting the thresholds lets us tune the classifications for resources and desired quality. If our humans doing data entry are particularly expensive or impatient, raising the thresholds should ensure that they are only very rarely sent typed text. On the other hand, lowering the thresholds would increase the human workload while improving quality of the resulting text.

One of the false negatives: T=0, N=10% from parsing terse and noisy text files.

I'm really pleased with this result. The combined classifications are slightly better than I was able to accomplish by looking at the OCR myself. The experience of a volunteer presented with 56 images containing handwriting and 13 which don't may necessitate a "send to OCR" button in the user interface, but must be less frustrating than the unclassified ratio of 45 in 105 from the sample set. With a different distribution of handwriting-to-type in the dataset, the process might be very useful for extracting rare typed material from a mostly-handwritten set, or vice versa.

All of the datasets, code, and scored CSV files are in iDigBio AOCR Hackathon's HandwritingDetection reposity on GitHub..

Results of the "Ocrocrop" Approach to Improving OCR

noreply@blogger.com (Ben W. Brumfield) — Fri, 15 Feb 2013 22:21:00 +0000

This project attempted to improve the quality of OCR applied to difficult entomology images[*] by cropping labels from the images to run through OCR separately. In order to identify labels on the image to crop, an initial, 'naive' pass of OCR was made over the whole image, generating both

A) a set of rectangles on the image defined as word bounding boxes by the OCR engine, and
B) a control OCR text file to be used for comparing the 'naive' model with the methodology.

Those word rectangles were then filtered, consolidated, and filtered again to identify the labels on the image, which were then extracted and run through the OCR engine separately. The resulting OCR output files were then concatenated into a single text file, which was compared against the 'naive' output described in A (above).

I'll call this method "ocrocrop". (For more detail on method, see the transcript of my preliminary presentation.)

The results were encouraging. (See CSV file listing results for each file, and the directory containing "naive" output, annotated JPGs, and cleaned output files for each test.)

Of 80 files tested, 20 experienced a decrease in score (see Alex Thomson's scoring service), but most (14/20) of those were on OCR output below 10% accuracy in the first place, and the remainder were at or below 20% accuracy. So it is reasonable to say that the ocrocrop method only degraded the quality of texts that were unusable in the first place.

40 of the 80 files tested showed more promising results, showing improvements from one to twenty percentage points -- in some cases only marginally improving unusable (below 10% accurate) outputs, but in many cases improving the scores more substantially (say from 25% to 35% in the case of EMEC609908_Stigmus_sp).

Most of the top quartile of results saw improvements on texts that were already scoring above 10% accuracy rates (16 of 20), so it appears that the effectiveness of the ocrocrop method is correlated to the quality of the naive input data -- garbage is degraded or only minimally improved, while OCR that is merely bad under the naive approach can be significantly improved.

The ocrocrop method saw the greatest improvement in cases where the naive OCR pass was effective at identifying word bounding boxes, but ineffective at translating their contents into words. Taking EMEC609928_Stigmus_sp, the case of greatest improvement (naive: 18.9%, ocrocrop: 70.5%), we see that all words on the labels except for the collector name were recognized as words (in purple), making the cropped label images (in blue) good representatives of the actual labels on the image.

The cropped image was more easily processed by our OCR image, so that we may compare the naive version of the second label:

 CALIF:Hunbo1dt Co. ;‘ ~
 3 m1.N' Garbervﬂle ,::f< '_- '
 v—23~75 n.n1e:z.' 9 ._ ’

with the ocrocrop version of the second label:

 CALIF:Humboldt Co.
 3 mi.N Garberville
 V-23-76 R.Dietz,'

One of the problems with the OCR-based pre-processing which may be hidden by the scores is that many labels are entirely missed by the ocrocrop if the first, naive OCR pass failed to identify any words at all on the label. In cases such as EMEC609651_Cerceris_completa, the determination label was not cropped (indicated by blue rectangles) because no words (purple rectangles) were detected by the original. As a result, while the ocrocrop OCR is an improvement over the naive OCR (6.6% vs. 6.5%), substantial portions of text on the image are unimproved because they are unattempted.

There are two possible ways to solve this problem. One is to abandon the ocrocrop model entirely, switching back to a computer vision approach -- either by programmatically locating rectangles on the image (as Phuc Nguyen demonstrated) or by asking humans to identify regions of interest for OCR processing (as demonstrated by Jason Best in Apiary and by Paul and Robin Schroeder in ScioTR). The other option is to improve the naive OCR -- perhaps by swapping out the engine (e.g. use ABBY instead of Tesseract), perhaps by using a different image pre-processor (like ocropus's front-end to Tesseract), perhaps by re-training Tesseract.

I suspect that a computer vision approach to extracting entomology labels (or similar pieces of paper photographed against a noisy background) will provide a more effective eventual solution than the ocrocrop method. Nevertheless, the ocrocrop "bang it with a rock until it works" approach has a lot of potential to take entomology-style OCR to bad from worse.

[*]In addition to the difficulties typical of specimen labels--mix of typefaces, handwritten material, typewritten material, text inventory with few overlaps with a dictionary of literary English--the entomology dataset contained additional challenges. Difficulties included the following:

Images containing specimens and rulers as well as labels.
Labels casually arranged for photography, so that text orientation was not necessarily aligned.
Labels photographed against a background of heavily pin-pricked styrofoam rather than a black or neutral background.
3-d images including what appear to be shadows, which soften the contrast differences around borders.

iDigBio Augmenting OCR Hackathon

noreply@blogger.com (Ben W. Brumfield) — Fri, 15 Feb 2013 22:00:00 +0000

I spent the last three days at the iDigBio Augmenting OCR Hackathon working alongside mycologists, botanists, entomologists, herbarium managers, and bioinformaticians to explore ways to improve parsing of digitized specimen labels. While I'm pleased with the results of my own contribution, I'd like to take a minute to talk about the hackathon process itself before I post them.

This was my first hackathon--a condition which seemed to be the rule among the participants--and I was really impressed with it. The iDigBio folks defined a clear set of goals (improve OCR parsing of specimen labels) with clear metrics (these datasets, these output formats, this scoring algorithm) a couple of months beforehand, and organized five weekly videoconferences before the event. Most important of all, the participants were encouraged to prepare a 10-minute lightning talk on their efforts and preliminary results. (See below for the transcript of my talk, see the notes document for descriptions of all talks.)

In my opinion, these preliminary talks were critical to the success of the project. The preliminary nature relaxed pressure on participants, so we were able to experiment beyond the target of the hackathon (as I did with my handwriting detection digression, a related, but un-scorable effort). On the other hand, they did provide enough impetus to get many of us looking at the data, working with the tools, and thinking about approaches. This meant that even before the hackathon started, many of us were familiar enough with the materials to have a real 'meeting of the minds' experience during the pre-event supper: "Did you just say 'the contrast difference between the print and the label is higher than the difference between the label and the background'? We ran into that too, and here's what we did..."

The experience was a real education in OCR for me, and I feel like I picked up techniques I can apply directly to projects I've discussed with clients and potential clients. In particular, I got a real appreciation for how interrelated image preparation, OCR, and parsing are to each other. One participant had created separate libraries of regular expressions to clean up each kind of field, having discovered that latitude/longitude coordinates require different error correction than personal names or herbarium catalog numbers do. Another group had built a touch-screen tool for classifying segments of the image before submitting them to OCR. My own project required a first pass of OCR to clean images before sending them to a second, 'real' pass of OCR. A simple 1,2,3 workflow just isn't sufficient!

iDigBio itself is an NSF-funded attempt to advance digitization practices on natural history collections, combining disciplinary "thematic collection networks" and methodologically focused working groups on topics like georeferencing, crowdsourcing, and OCR. Aware that they're not the only people digitizing things, they have been reaching out beyond the natural sciences to the library and information science community at the iConference this year. This rejection of "not invented here" siloing was a big part of the hackathon, and I hope that more people from outside the natural sciences will get involved.

Improving OCR Inputs from OCR Outputs?

noreply@blogger.com (Ben W. Brumfield) — Thu, 14 Feb 2013 15:32:00 +0000

This is a transcript of my talk at the iDigBio Augmenting OCR Hackathon, presenting preliminary results of my efforts before the event.

For my preliminary work, I tried to improve the inputs to our OCR process through looking at the outputs of a naive OCR.

One of the first things that we can do to improve the quality of our inputs to OCR is to not feed them handwriting. To quote Homer Simpson, "Remember son, if you don't try, you can't fail." So let's not try feeding our OCR processes handwritten materials.

To do this, we need to try to detect the presence of handwriting. When you try to feed handwriting to OCR, you get a lot of gibberish. If we can detect handwriting, we can route some of our material to "humans in the loop" -- not wasting their time with things we could be OCRing. So how do we do this?

My approach was to use the outputs of [naive] OCR to detect the gibberish it produces when it sees handwriting to try to determine when there was handwriting present in the images. The first thing I did before I started programming, was classifying OCR output from the lichen samples by visual inspection: whether I thought there was hand writing present or not, based on looking at the OCR outputs. Step two was to automate the classifications.

I tried this initially on the results that came out of ABBY and then the results that came out of Tesseract, and I was really surprised by how hard it was for me as a human to spot gibberish. I could spot it, but in a lot of cases -- ABBY does a great job of cleaning up its OCR output -- so in a lot of cases, particularly the labels that were all printed with the exception of some species name that was handwritten, ABBY generally misses those. Tesseract, on the other hand, does not produce outputs that are quite as clean.

So the really interesting thing about this to me is that while we were able to get 70-75% accuracy on both ABBY and Tesseract, if you look at the difference between the false positives that come out of ABBY and Tesseract and the false negatives, I think there is some real potential here for making a much more sophisticated algorithm. Maybe the goal is to pump things through ABBY for OCR, but beforehand look at Tesseract output to determine whether there is handwriting or not.

The next thing I did was try to automate this. I just used some regular expressions to look for representative gibberish, and then based on the number of matches got results that matched the visual inspection, though you do get some false positives.

The next thing I want to do with this is to come up with a way to filter the results based on doing a detection on ABBY [output] and doing a detection on Tesseract [output].

The next thing that I wanted to work on was label extraction.

We're all familiar with the entomology labels and problems associated with them.

So if you pump that image of Cerceris through Tesseract, you end up with a lot of garbage. You end up with a lot of gibberish, a lot of blank lines, some recognizable words. That "Cerceris compacta" is, I believe, the result of a post-digitzation process: it looks like an artifact of somebody using Photoshop or ImageMagick to add labels to the image. The rest of it is the actual label contents, and it's pretty horrible. We've all stared at this; we've all seen it.

So how do you sort the labels in these images from rulers, holes in styrofoam, and bugs? I tried a couple of approaches. I first tried to traverse the image itself, looking for contrast differences between the more-or-less white labels and their backgrounds. The problem I found with that was that the highest contrast regions of the image are the difference between print and the labels behind the print. So you're looking for a fairly low-contrast difference--and there are shadows involved. Probably, if I had more math I could do this, but this was too hard.

So my second try was to use the output of OCR that produces these word bounding boxes to determine where labels might be, because labels have words on them.

If you run Tesseract or Ocropus with an "hocr" option, you get these pseudo-HTML files that have bounding boxes around the text. Here you see this text element inside a span; the span has these HTML attributes that say "this is an OCR word". Most importantly, you have the title attribute as the bounding box definition of a rectangle.

If you extract that and re-apply it to an image, you see that there are a lot of rectangles on the image, but not all the rectangles are words. You've got bees, you've got rulers; you've got a lot of random trash in the styrofoam.

So how do we sort good rectangles from bad rectangles? First I did a pass looking at the OCR text itself. If the bounding box was around text that looked like a word, I decided that this was a good rectangle. Next, I did a pass by size. A lot of the dots in the stryofoam come out looking suspiciously word-like for reasons I don't understand. So if the area of the rectangle was smaller than .015% of the image, I threw it away.

The result was [above]: you see rectangles marked with green that pass my filter and rectangles marked with red that don't. So you get rid of the bee, you get rid of part of the ruler -- more important, you get rid of a lot of the trash over here. [Pointing to small red rectangles on styrofoam.] There are some bugs in this--we end up getting rid of "Arizona" for reasons I need to look at--but it does clean the thing up pretty nicely.

Question: A very simple solution to this would be for the guys at Berkeley to take two photographs -- one of the bee and ruler, one of the labels. I'm just thinking how much simpler that would be.

Me: If the guys in Berkeley had a workflow that took the picture--even with the bee--agaist a black background, that would trivialize this problem completely!

Question: If the photos were taken against a background of wallpaper with random letters, it couldn't be much worse than this [styrofoam]. The idea is that you could make this a lot easier if you would go to the museums and say, we'll participate, we'll do your OCRing, but you must take photographs this way.

Me: You're absolutely right. You could even hand them a piece of cardboard that was a particular color and say, "Use this and we'll do it for you, don't use it and we won't." I completly agree. But this is what we're starting with, so this is what I'm working on.

The next thing is to aggregate all those word boxes into the labels [they constitute]. For each rectangle, look at all of the other rectangles in the system, expand them both a little bit, determine if they overlap, and if they do, consolidate them into a new rectangle, and repeat the process until there are no more consolidations to be done. [Thanks to Sara Brumfield for this algorithm.]

If you do that, the blue boxes are the consolidated rectangles. Here you see a rectangle around the U.C. Berkeley label, a rectangle around the collector, and a pretty glorious rectangle around the determination that does not include the border.

Having done that, you want to further filter those rectangles. Labels contain words, so you can reject any rectangles that were "primitives" -- you can get rid of the ruler rectangle, for example, because it was just a single [primitive] rectangle that was pretty large.

So you make sure that all of your rectangles were created through consolidation, then you crop the results. And you end up automatically extracting these images from that sample -- some of which are pretty good, some of which are not. We've got some extra trash here, we cropped the top of "Arizona" here. But for some of the labels -- I don't think I could do better than that determination label by hand.

Then you feed the results back into Tesseract one by one, then we combine the text files in Y-axis order to produce a single file for all those images. (Not something that's a necessary step, but that does allow us to compare the results with the "raw" OCR.) How did we do?

This is a resulting text file -- we've got a date that's pretty recognizable, we've got a label that's recognizable, and the determination is pretty nice.

Let's compare it to the raw result. In the cropped results, we somehow missed the "Cerceris compacta", we did a much nicer job on the date, and the determination is actually pretty nice.

Let's try it on a different specimen image.

We run the same process over this Stigmus image. We again find labels pretty well.

When we crop them out, the autocrop pulls them out into these three images.

Running those images through OCR, we get a comparison of the original, which had a whole lot of gibberish.

The original did a decent job with the specimen number, but the autocrop version does as well. In particular, for this location [field], the autocrop version is nearly perfect, whereas the original is just a mess.

My conclusion is that we can extract labels fairly effectly by first doing a naive pass of OCR and looking at the results of that, and that the results of OCR over the cropped images is less horrible than running OCR over the raw images -- though still not great.

[2013-02-15 update: See the results of this approach and my write-up of the iDigBio Augmenting OCR Hackathon itself.]

What does it mean to "support TEI" for manuscript transcription?

noreply@blogger.com (Ben W. Brumfield) — Sat, 10 Nov 2012 14:57:00 +0000

This is a transcript of my talk at the 2012 TEI meeting at Texas A&M University, "What does it mean to 'support TEI' for manuscript transcription: a tool-maker's perspective."

You can download an MP3 recording of the talk here.

Let's get started with a couple of definitions. All the tools and the sites that I'm reviewing are cloud based, which means that I'm ruling out--perhaps arbitrarily--any projects that involve people doing offline edition and then publishing that on the web. I'm only talking about online-based tools.

So that's a very strict definition of clouds, and I'm going to have a very loose and squishy definition of crowds, in which I'm talking about any sort of tool that allows collaborative editing of manuscript material, and not just ones that are directed at amateurs. That's important for a couple of reasons: one, because it gave me a sample size that was large enough to find out how people are using TEI, but--for another reason--because "amateurs" aren't really amateurs. What we see with crowdsourcing projects is that amateurs become experts very quickly. And given that your average user of any citizen science or historical crowdsourcing project is a woman over 50 who has at least a Master's degree, this isn't sort of the unwashed masses.

Okay, so crowdsourced transcription has been going on for a while, and it's been happening in four different traditions that developed this all independently. You have genealogists who are doing this, primarily with things like census records. The 1940 census is the most prominent example: they have volunteers transcribing as many as ten million records a day. The natural sciences are doing something similar, particularly GalaxyZoo, the OldWeather people are looking at climate change data, where you have to look at old, handwritten records to figure out how the climate has changed, because you need to know how the climate used to be. And then there are also some projects going on in the Open Source/Creative Commons world: the Wikisource people--particularly the German language Wikisource community--and libraries, archives, and museums have jumped into this recently.

So here are a couple of examples from the citizen science world. OldWeather has a tool that allows people to record ship log book entries and weather observations. As you can see, this is all field based -- this isn't quite an attempt to represent a document. We'll get back to this in a minute.

The North American Bird Phenology Program is transcribing old bird[-watching] observation cards from about a hundred years ago. They're recording species names and all sorts of other things about this particular Grosbeak in 1938.

All of these--this is the majority of the crowdsourced transcription that's happening out there--there are millions of records--there are millions of records that are happening that are all record based. These are not document-based, they aren't page-based. They're dealing with data that is fundamentally tabular -- those are their inputs. Their outputs are databases that they want to be able to either search or analyze. So we're producing nothing that anyone would ever want to print out.

And another interesting thing about this is that these record-based transcription projects--the uses are understood in advance. If you're building a genealogy index, you know that people are going to want to search for names and be able to see the results. And that's it -- you're not building something that allows someone to go off and do some other kind of analysis.

Now what kind of mark-up are these record-based transcription projects using? Well, it's kind of idiosyncratic, at best.

Here's an example from my client FreeREG. This is a mark-up language that they developed about ten years ago for indicating unclear readings of manuscripts. It's actually fairly sophisticated--it's based on the regular expression programming sub-language--but it's not anything that's informed by the TEI world.

On the other hand, here is the mark-up that the New York Public Library is using. Let me read this out to you: "Please type the text of the indicated dish exactly as it appears. Don't worry about accents." This is almost an anti-markup.

So what about free-form transcription? There's a lot of development of people doing free-form transcription. You have Scripto out of CHNM. You have a couple of different (perhaps competing) NARA initiatives. Wikisource. There's my own FromThePage. What kind of mark-up are they doing? Well, for the most part, none!

Here's Scripto--the Papers of the War Department-- and you type what you see, and that's what you get.

Here is the French-language Wikisource, hosting materials from the Archives departmentales du Cantal (who are doing some very cool things here). But this is just typing things into a wiki and not even internally using wiki links. This is almost pre-formed text -- it's pretty much plaintext.

My own project, FromThePage.

I'm internally using wiki-links, but really only for creating indexes and annotations, not for indicating...any of the power that you have with TEI.

So if no one is using TEI, why is TEI important? I think that TEI is important because crowdsourced transcription projects are how the public is interacting with edition. This is how people are learning what editing is, what the editing process is, and why and whether it's important. And they're using tools that are developed by people like me. Now how do people like me learn about edition?

The answer is, by reading the TEI Guidelines. The TEI Guidelines have an impact that goes far beyond people who are actually implementing TEI. I started work on FromThePage in complete isolation in 2005. By 2007, I was reading the TEI Guidelines. I wasn't implementing TEI, but the questions that were asked--these notions of "here's how you expand abbreviations", "here's how you regularize things"--had a tremendous impact on me. By contrast, the Guide to Documentary Editing--which is a wonderful book!--I only found out in January of this year.

TEI is online, it's concise, it's available. And when I talk to people in the genealogy development world, they know about TEI. They've heard of it. They have opinions. They're not using it, but -- you people are making an impact on how the world does edition!

Okay, so if all of these people aren't using TEI, who is doing it?

I run a transcription tool directory that is itself crowdsourced. It's been edited by 23 different people who've entered information about 27 different tools. Of those 27 tools, 7 are marked as "supporting TEI". There's a little column, "does it support TEI?", seven of them say "Yes".

Actually, that's not true. Some of them say "yes", but some of those seven say "well, sort of". So what does that mean?

To find that out, I interviewed five of those seven projects.

Transcribe Bentham.
T-PEN (which there's a poster session about tonight), which is a line-based system for medieval manuscripts.
A customization of T-PEN, the Carolingian Canon Law project, out of the University of Kentucky.
Our own Hugh Cayless for the Papyrological Editor, which is dealing with papyri.
And then MOM-CA is one of these "sort of"s. You have two implementations of it.

One of them is the Virtualles deutsches Urkundennetzwerk, which is a German charter collection. It supports "TEI, sort-of" -- actually it supports CEI and EAD.
But it's been customized for extensive TEI support for the Itinera Nova project which is out of the archive of Leuven, Belgium.

I'm going to talk about what I found out, but I'm going to emphasize Transcribe Bentham. Not because it's better than the other tools, but because they actually ran their transcription project as an experiment. They wanted to know, can the public do TEI? Can the public handle it? And they've published their results: they've conducted user surveys of what was your experience using TEI? Which makes it particularly useful for those of us who are trying to figure out how it's being used.

Okay, so there's a lot of variation among these projects. You've got a varied committment to TEI. Transcribe Bentham: Yes, we're going to use TEI! You see Melissa Terras here saying that "it was untenable" that we'd ask for anything else. These people know how to do it; why would we depart from that?

For T-PEN, James Ginther says: Hey, I'm kind of skeptical. We'll support any XSD you want to upload, if it happens to be TEI, that's okay.

Abigail Firey, who's using T-PEN, basically says: look, it's probably necessary. It's very useful. It lets us develop these valuable intellectual perspectives on our text. And she considered it important that their text encoding was done within the community of practice represented by the people in this room.

Okay, so more variation between these. Where's the TEI located within these projects? Where does it live? I'm a developer; I'm interested in the application stack.

It turns out that there's no agreement at all. Transcribe Bentham has people entering TEI in person. And then it's storing it off in a MediaWiki, using MediaWiki versioning, not actually putting [...] pages in one big TEI document.

On the other hand, Itinera Nova is actually storing everything in an XRX-based XML database. I mean, it is pure TEI on the back end. But none of the volunteers using Itinera Nova actually are typing any angle brackets. So we have a lot of variation here.

However, there was no variation when I asked people about encoding. There is a perfectly common perception that is: Encoding is hard!

And there are these great responses--that you can see both on the Transcribe Bentham blog and in their DHQuarterly paper that just came out, which I highly recommend--describing it as "too much markup", "unnecessarily complicated", "a hopeless nightmare", and the entire transcription process is "a horror."

But, lots of things are hard.

In my own experience with FromThePage, I have one user who has transcribed one thousand pages, but she does not like using any mark-up at all. She's contributing! She's contributing plaintext transcriptions, but I'm going back to add wikilinks. So it's not about the angle brackets. (Maybe square brackets have a problem too, I don't know.)

And fundamentally, transcribing--reading old manuscripts--is hard. "Deciphering Bentham's hand took longer than encoding," for over half of the Bentham respondents.

So there's more commonality: everyone wants to make encoding easier. How do we do that? There's a couple of different approaches. One approach--the most common approach--is using different kinds of buttons and menus to automate the insertion of tags. Which gets around (primarily) the need for people to memorize tag names and attributes, and--God help us--close tags.

So these are implemented--we've got buttons on T-PEN and CCL. We've got buttons on the TEI Toolbar. We've got menus on VdU and the Papyrological Editor.

And you can see them. Here's a screenshot of Jeremy Bentham. A couple of interesting things about this: it's very small, but we've got a toolbar at the top. We've got TEI text: angle-bracket D.E.L. Angle-bracket, slash, D.E.L. So we're actually exposing the TEI to users in Transcribe Bentham, though we're providing them with some buttons.

Those buttons represent a subset--I'll get to the selection of those tags later. Here's a more detailed description of what they do.

Here's what's going on with VdU. Only in this case, they're not actually exposing the angle brackets to the user. They're replacing all of these in a pseudo-WYSIWYG that allows people to choose from a menu and select text that then gets tagged.

Okay -- limitations of the buttons. There's a good limitation, which is that as users become more comfortable with TEI, they outgrow buttons. And this is something that the people at Transcribe Bentham reported to me. They're seeing a fair number of people just skip the buttons altogether and type angle brackets. Remember: these are members of the public who have never met any of the Transcribe Bentham people.

On the down side, users also ignore the buttons. Again users ignoring encoding, but in this case we've got something that's a little bit worse. Georg Vogeler is reporting something very interesting, which is that in a lot of cases, they were seeing users who were using print apparatus for doing this kind of work, and just ignoring the buttons -- going around them.

So the problem with using print-style notations. People are dealing with these print editions [notations] -- this can be a problem or it can be an opportunity. Papyri.info is viewing it that way. Itinera Nova is using it that way.

Papyri.info, their front-end interface for most users is Leiden+, which is a standard for marking up papyri. And, as you can see, users enter text in Leiden+, and that generates TEI. (EpiDoc TEI, I believe.)

This is the same kind of process that's done in Itinera Nova. In that case, they're using for notation whatever it is that the Leuven archives uses for their mark-up. And they're doing the same kind of transposition [ed: translation] of replacing their notation with TEI tags before they save it.

And this is actually what users see as they're typing. They don't see the TEI tags -- we're hiding the angle brackets from them.

So this is an alternative to buttons. And in my opinion, it's not that bad an alternative.

This hasn't been a problem for the Bentham people, however. It's a non-problem for them. And they are the most "crowdy", the most amateur-focused, and the most committed to a TEI interface.

Tim Causer went through and reviewed all of this and said, you know, it just doesn't happen. People are not using any print notation at all. They're using buttons. They're using angle-brackets by hand. They're not even using plaintext. They're using TEI. Their users are comfortable with TEI.

So what accounts for the difference between the experience of the VdU and the Transcribe Bentham people? I don't know. I've got a couple of theories about what might be going on.

One of them is really the corpus of texts we're working with. If you're only dealing with papyrus fragments, and you're used to a well-established way of notating them--that's been around since 1935 in the case of Leiden+--well, it's kind of hard to break out of that. On the other hand, there's not a single convention for print editions. There's all sorts of ways of indicating additions and deletions for print editions of more modern texts. So maybe it's a lack of a standard.

Or, maybe it's who the users are. Maybe scholars are stubborner, and amateurs are more tractable and don't have bad habits to break. I don't know! I don't know, but I'd be really interested in any other ideas.

Okay, how do these projects choose the tags that they're dealing with? We've got a very long quote, but I'm just going to read out a couple of little bits of them.

Really, choosing a subset of tags is important. Showing 67 buttons was not a good usability thing for T-PEN. And in particular, what they ended up doing was getting rid of the larger, structural set of markup, and focusing just on sort of phrase-level markup.

This also, I think, true if we go back a minute and look at Bentham. Here, again, we're talking phrase-level tags. We're not talking about anything beyond that.

Justin Tonra said that it was actually really hard to pare down the number of tags for Transcribe Bentham. He wanted to do more, but, you know, he's pleased with what they got. They didn't want to "overcomplicate the user's job."

Richard Davis, also with Transcribe Bentham, had a great deal of experience dealing with editors for EAD and other XML. And he said you're always dealing with this balance between usability and flexibility, and there's just not much way of getting around it. It's going to be a compromise, no matter what.

So what's the future for these projects that are using TEI for crowds? Well, if getting people up to speed is hard, and if nobody reads the help--as Valerie Wallace at one time said about their absolutely intimidating help page for Transcribe Bentham (you should look at it -- it's amazing!)--then what are the alternatives for getting people up to speed?

Georg Vogeler says that they are trying to come up with a way of teaching people how to use the tool and how to use the markup in almost a game-like scenario. We're not talking about the kind of Whak-a-Mole things that we sometimes see, but really just sort of leading people through Let's try this. Now let's try this. Now let's try this. Okay now you know how to deal with this [tool]. It's something that I think we're actually pretty familiar with from any other kinds of projects dealing with historic handwriting.: people have to come up to speed.

Another possibility is a WYSIWYG. Tim Causer announced the idea of spending their new Mellon grant on building a WYSIWYG for Transcribe Bentham's TEI. The blog entry is fascinating because he gets about seven user comments, some of which express a whole lot of skepticism that a WYSIWYG is going to be able to handle nested tagging in particular. Other ones of which make comments about the whole XML system and its usability in vivid prose, which is very worth reading.

And maybe combinations of these. So we have these intermediate notations -- Itinera Nova, for example, they're using this let's begin a strike-through with an equals sign (which is apparently what they've been using at that archive for a while). And the minute you type that equals sign in, you actually get a WYSIWYG strike-through that runs all the way through your transcript.

That may be the future. We'll see. I think that we have a lot of room for exploring different ways for handling this.

So let me wrap up and thank my interviewees.

Transcribe Bentham: Melissa Terras, Justin Tonra, Tim Causer, Richard Davis.

T-PEN: James Ginther, Abigail Firey
Papyri.info: Hugh Cayless, Tom Elliot
MOM-CA: Georg Vogeler and Jochen Graf

Questions

[All questions will be paraphrased in the transcript due to sound quality, and are not to be regarded as direct quotations without verification via the audio.]

Syd Bauman: Of the systems which allow users to type tags free-hand, what percentage come out well-formed?

Me: The only one that presents free-hand [tagging] is Transcribe Bentham. Tim [Causer] gets well-formed XML for most everything he gets. There is no validation being performed by that wiki, but what he's getting is pretty good. He says that the biggest challenge when he's post-processing documents is closing tags and mis-placed nesting.

Syd Bauman: I'd be curious about the exact percentages.

Me: Right. I'd have to go back and look at my interview. He said that it represents a pretty small percentage, like single digits of the submissions they get.

John Unsworth: Do any of the systems use keyboard short-cuts?

Me: I know of none that use hot-keys.

John Unsworth: Do you think that would be more or less desirable than the systems you've described?

Me: I really only see hot-keys as being desirable for projects that are using more recent and clearer documents. Speed of data-entry from the keyboard perspective doesn't help much when you're having to stare and zoom and scroll on a document that is as dense and illegible as Bentham or Greek papyri.

Elena Pierazzo [very faint audio]: In some cases it's hard to define which is the error: choosing the tags or reading the text. I've been working with my students on Transcribe Bentham--they're all TEI-aware--and to be honest it was hard. The difficulty was not the mark-up. In a sense we do sometimes forget in these crowdsourcing projects, that the text itself is very hard, so probably adding a level of complexity to the task via the mark-up is very difficult.

I have all respect and sympathy for the people who stick to the ideal of doing TEI, which I commend entirely. But in some cases, it may be that asking amateur people to do [the decipherment] and do the mark up is a pretty strong request, and makes a big assumption about what the people "out there" are capable of without formation.

Me: I'd agree with you. However, there have been some studies on these users' ability to produce quality transcripts outside of the TEI world.... Old Weather did a great deal of research on that, and they found that individual users tended to submit correct transcripts 97% of the time. They're doing blind triple-keying, so they're comparing people's transcripts against others. [They found] that of 1000 different entries, typically on average 13 will be wrong. Of those thirteen, three will be due to user error--so it does happen; I'm not saying people are perfect. Three will be generally[ed: genuinely] illegible. And the remaining seven will be due to the officer of the watch having written the wrong thing down and placing the ship in Afghanistan instead of in the Indian Ocean. So there are errors everywhere. [I mis-remembered the numbers here: actually it's 3 errors due to transcriber error, 10 genuinely illegible, and 3 due to error at time of inscription.]

Lou Burnard: The concept of error is a nuanced one. I would like to counter-argue Elena's [point]. I think that one of the reasons that Bentham has been successful is precisely because it's difficult material. Why do I think that? Because if you are faced with something difficult, you need something powerful to express your understanding of it. The problem with not using something as rich and semantically expressive as TEI when you're doing your transcription is that it doesn't exist! All you can do is type in the words you think it might have been, and possibly put in some arbitrary code to say, "Well, I'm not sure about that." Once you've mastered the semantics of the TEI markup--which doesn't actually take that long, if you're interested in it--now you can express yourself. Now you can communicate in a [...] satisfactory way. And I think that's why people like it.

Me: I have anecdotal, personal evidence to agree with you. In my own system (that does not use TEI), I have had users who have transcribed several pages, and then they'd get to a table in some biologist's field notes, for example, and they stop. And they say, "well, I don't know what to do here." So they're done.

Lou Burnard: The example you cite of the erroneous data in the source is a very good one, because if you've mastered TEI then you know how to express in markup: 'this is what it actually says but clearly he wasn't in Afghanistan.' And that isn't the case in any other markup system I've ever heard of.

[I welcome corrections to my transcript or the contents of the talk itself at benwbrum@gmail.com or in the comments to this post.]

Interview with Ben Crowder on Unbindery

noreply@blogger.com (Ben W. Brumfield) — Wed, 24 Oct 2012 15:10:00 +0000

One of the pleasures of maintaining the crowdsourced transcription tool list is learning about systems I'd never heard about before. One of these is Unbindery, a tool being built by Ben Crowder for audio and manuscript transcription as well as OCR correction. Ben was gracious enough to grant me an interview, even though he's concentrating on the final stretch of development work on Unbindery.

First, let me wish you luck as you enter the final push on Unbindery. What would you say is the most essential feature you have left to work on?

Thanks! Probably private projects -- I've been looking forward to using Unbindery to transcribe my journals, but haven't wanted them to be open for just anyone to work on. I'm also very excited about chunking audio into small segments (I used to publish an online magazine where we primarily published interviews, and transcribing two hours of audio can be really daunting).

Tell us more about how Unbindery handles both audio transcription and manuscript transcription. Usually those tools are very different, aren't they?

The audio transcription part started out as Crosswrite, a little proof-of-concept I threw together when I realized one day that JavaScript would let me control the playhead on an audio element, making it really easy to write a software version of a transcription foot pedal. I also wanted to start using Unbindery for family history purposes (transcribing audio interviews with my grandparents, mainly, and divvying up that workload among my siblings).

So, to handle both audio transcription and page image transcription, Unbindery has a modular item type editor system. Each item type has its own set of code (HTML/CSS/JavaScript) that it loads when transcribing an item. For example, page images show an image and a text box, with some JavaScript to place a highlight line when you click on the image, whereas audio items replace the image with Crosswrite's audio element (and the keyboard controls for rewinding and fast forwarding the audio). It would be fairly trivial to add, say, an item type editor that lets the user mark up parts of the transcript with XML tags pulled from a database or web service somewhere. Or an editor for transcribing video. It's pretty flexible.

How did you come up with the idea for Unbindery?

I had done some Project Gutenberg work back in 2002, and somewhere along the way I came across Distributed Proofreaders, which basically does the same thing. A few years later, I'd recently gotten home from an LDS mission to Thailand and wanted to start a Thai branch of Project Gutenberg with one of my mission friends. He came up with the name Unbindery and I made some mockups, but nothing happened until 2010 when I launched my Mormon Texts Project. Manually sending batches of images and text for volunteers to proof was laborious at best, so I was motivated to finally write Unbindery. I threw together a prototype in a couple weeks and we've been using it for MTP ever since. I'm also nearing the end of a complete rewrite to make Unbindery more extensible and useful to other people. And because the original code was ugly and nasty and seriously embarrassing.

In my experience, the transcription tools that currently exist are very much informed by the texts they were built to work with, with some concentrating on OCR-correction, others on semantic indexing, and others on mark-up of handwritten changes to the text. How do you feel like the Mormon Texts Project has shaped the features and focus of Unbindery?

Mormon Texts Project has been entirely focused on correcting OCR for publication in nice, clean ebook editions, which is why we've gone with a plain old text box and not much more than that. (Especially considering that we were originally posting the books to Project Gutenberg, where our target output format was very plain text.)

What is your grand dream for Unbindery? (Feel free to be sweeping here and assume grateful, enthusiastic users and legions of cobbler's elves to help with the code.)

To get men on Mars. No, really, I don't think my dreams for Unbindery are all that grand -- I'd be more than satisfied if it helps make transcription easier for users, whether working alone or in groups, and whether they're publishing ebooks or magazines or transcribing oral histories or journals or what have you.

In an ideal world it would be wonderful if a small, dedicated group of coders were to adopt it and take care of it going forward. But I don't expect that. I'll get it to a state where I can publicly release it and people can use it, but other than bugfixes, I don't see myself doing much active development on Unbindery beyond that point. I know, I know, abandoning my project before it's even out the door makes me a horrible open source developer. But to be honest with you, I don't really even want to be an open source developer -- I'm far more interested in my other projects (like MTP) and I want to get back to doing those things. Unbindery is just a tool I needed, an itch I scratched because there wasn't anything out there that met my needs. People have expressed interest in using it so I'm putting it up on GitHub for free, but I don't see myself doing much with Unbindery after that. Sorry! This is the sad part of the interview.

What programming languages or technical frameworks do you work in?

Unbindery is PHP with JavaScript for the front end. I love JavaScript, but I'm only using PHP because of its ubiquity -- I'd much, much, much rather use Python. But it's a lot easier for people to get PHP apps running on cheap shared hosts, so there you have it.

It seems like you're putting a lot of effort into ease of deployment. How do you see Unbindery being used? Do you expect to offer hosting, do you hope people install their own instances, or is there another model you hope to follow?

I won't be offering hosting, so yes, I'm expecting people to install their own instances, and that's why I want it to be easy to install. (There may be some people who decide to offer hosting for it as well, and that's fine by me.)

How can people get involved with the project?

Coders: The code isn't quite ready for other people to hack on it yet, but it's getting a lot closer to that point. For now, coders can look at my roadmap page to see what tasks need doing. (Also, it won't be long before I start adding issues to GitHub so people can help squash bugs.)

Other people: Once the core functionality is in place, just having people install it and test it would probably be the most helpful.

Jens Brokfeld's Thesis on Crowdsourced Transcription

noreply@blogger.com (Ben W. Brumfield) — Thu, 18 Oct 2012 13:51:00 +0000

Although the field of transcription tools has become increasingly popular over the last couple of years, most academic publications on the topic focus on a single project and the lessons that project can teach. While those provide invaluable advice on how to run crowdsourcing projects, they do not lend much help to memory professionals trying to decide which tools to explore when they begin a new project. Jens Brokfeld's thesis for his MLIS degree at Fachhochschule Potsdam is the most systematic, detailed, and thorough review of crowdsourced manuscript transcription tools to date.

After a general review of crowdsourcing cultural heritage, Brokfeld reviews Rose Holley's checklist for crowdsourcing projects and then expands upon the part of my own TCDL presentation which discussed criteria for selecting transcription tools, synthesizing it with published work on the subject. He then defines his own test criteria for transcription tools, about which more below. Then, informed by seventy responses to a bilingual survey of crowdsourced transcription users, Brokfeld evaluates six tools (FromThePage, Refine!, Wikisource, Scripto, T-PEN, and the Bentham Transcription Desk) with forty-two pages (pp. 40-82) devoted to tool-specific descriptions of the capabilities and gaps within each system. This exploration is followed by an eighteen-page comparison of the tools against each other (pp. 83-100). The whole paper is very much worth your time, and can be downloaded at the "Masterarbeit.pdf" link here: "Evaluation von Editionswerkzeugen zur nutzergenerierten Transkription handschriftlicher Quellen".

It would be asking too much of my limited German to translate the extensive tool descriptions, but I think I should acknowledge that I found no errors in Brokfeld's description of my own tool, FromThePage, so I'm confident in his evaluation of the other five systems. However, I feel like I ought to attempt to abstract and translate some of his criteria for evaluation, as well as his insightful analysis of each tool's suitability for a particular target group.

Chapter 5: Prüfkriterien ("Test Critera")

5.1 Accessibility (by which he means access to transcription data from different personal-computer-based clients)
5.1.1 Browser Support
5.2 Findability
5.2.1 Interfaces (including support for such API protocols as OAI-PMH, but including functionality to export transcripts in XML or to import facsimiles)
5.2.2 References to Standards (this includes support for normalization of personal and place names in the resulting editions)
5.3 Longevity
5.3.1 License (is the tool released under an open-source license that addresses digital preservation concerns?)
5.3.2 Encoding Format (TEI or something else?)
5.3.3 Hosting
5.4 Intellectual Integrity (primarily concerned with support for annotations and explicit notation of editorial emendations)
5.4.1 Text Markup
5.5 Usability (similar to "accessibility" in American usage)
5.5.1 Transcription Mode (transcriber workflows)
5.5.2 Presentation Mode (transcription display/navigation)
5.5.3 Editorial Statistics (tracking edits made by individual users)
5.5.4 User Management (how does the tool balance ease-of-use with preventing vandalism?)

I don't believe that I've seen many of these criteria used before, and would welcome a more complete translation.

His comparison based on target group is even more innovative. Brokfeld recognizes that different transcription projects have different needs, and is the first scholar to define those target groups. Chapter 7 of his thesis defines those groups as follows:

Science: The scientific community is characterized by concern over the richness of mark-up as well as a preference for customizability of the tool over simplicity of user interface. [Note: it is entirely possible that I mis-translated Wissenschaft as "science" instead of "scholarship".]
Family History: Usability and a simple transcription interface are paramount for family historians, but privacy concerns over personal data may play an important role in particular projects.
Archives: While archives attend to scholarly standards, their primary concern is for the transcription of extensive inventories of manuscripts -- for which shallow markup may be sufficient. Archives are particularly concerned with support for standards.
Libraries: Libraries pay particular attention to bibliographical standards. They also may organize their online transcription projects by fonds, folders, and boxes.
Museums: In many cases museums possess handwritten sources which refer to their material collections. As a result, their transcriptions need to be linked to the corresponding object.

It's very difficult for me to summarize or extract Brokfeld's evaluation of the six different tools for five different target groups, since those comparisons are in tabular form with extensive prose explanations. I encourage you to read the original, but I can provide a totally inadequate summary for the impatient:

FromThePage: Best for family history and libraries; worst for science.
Refine!: Best for libraries, followed by archives; worst for family history.
Wikisource: Best for libraries, archives and museums; worst for family history.
Scripto: Best for museums, followed by archives and libraries; worst for family history and science.
T-PEN: Best for science.
Bentham Transcription Desk: Best for libraries, archives and museums.

Note: This is a summary of part of a 140-page German document translated by an amateur. Consult the original before citing or making decisions based on the information here. Jens Brokfeld welcomes questions and comments (in English or German) through this webform: http://opus4.kobv.de/opus4-fhpotsdam/frontdoor/mail/toauthor/docId/331.

Webwise Reprise on Crowdsourcing

noreply@blogger.com (Ben W. Brumfield) — Wed, 10 Oct 2012 14:30:00 +0000

Back in June, the folks at IMLS and Heritage Preservation ran a webinar exploring the issues and tools discussed at the IMLS Webwise Crowdsourcing panel "Sharing Public History Work: Crowdsourcing Data and Sources."

After a introduction by Kevin Cherry and Kristen Laise, Sharon Leon, who chaired the live panel, presented a wonderful overview of crowdsourcing cultural heritage and discussed the kinds of crowdsourcing projects that have been successful -- including, of course, the Papers of the War Department and Scripto, the transcription tool the Roy Rosenzweig Center for History and New Media developed from that project. They then ran the video of my own presentation, "Lessons from Small Crowdsourcing Projects", followed by a live demo of FromThePage. Perhaps the best part of the webinar, however, was the Q&A from people all over the country asking for details about how these kinds of projects work.

The recording of the webinar is online, and I encourage you to check it out. (Here's a direct link, if you have trouble.) I'm very grateful to IMLS and Heritage Preservation for their work in making this knowledge accessible so effectively.

Mosman 1914-1918 on FromThePage

noreply@blogger.com (Ben W. Brumfield) — Tue, 09 Oct 2012 15:49:00 +0000

The Mosman community in New South Wales is preparing for the centennial of World War One, and as part of this project they've launched mosman1914-1918.net: "Doing our bit, Mosman 1914–1918". The project describes itself as "an innovative online resource to collect and display information about the wartime experiences of local service people," and includes scan-a-thons, hack days, and build-a-thons.

One of their efforts involves transcription of a serviceman's diary with links to related names on local honor boards. I'm delighted to report that they're hosting this project on FromThePage.com, and I look forward to working with and learning from the Mosman team.

Read more about Allan Allsop's diary on the Mosman 1914-1918 project blog and lend a hand transcribing!

Building a Structured Transcription Tool with FreeUKGen

noreply@blogger.com (Ben W. Brumfield) — Wed, 03 Oct 2012 14:13:00 +0000

I'm currently working with FreeUKGen--the charity behind the genealogy database FreeBMD--to build a general-purpose, open-source tool for crowdsourced transcription of structured manuscript data into a searchable database.

We're basing our system on the Scribe tool developed for the Citizen Science Alliance for What's the Score at the Bodleian, which originated out of their experience building OldWeather and other citizen science sites.

We are building the following systems:

A new tool for loading image sets into the Scribe system and attaching them to data-entry templates.
Modifications to the Scribe system to handle our volunteer organization's workflow, plus some usability enhancements.
A publicly-accessible search-and-display website to mine the database created through data entry.
A reporting, monitoring, and coordinating system for our volunteer supervisors.

We also plan to add support for geocoding during transcription and GIS support within the search and display system. Currently, initial development is mostly finished with 1 and moving on to 2 and 3 above.

Although this tool is focused on support for parish registers and census forms, we are intent on creating a general-purpose system for any tabular/structured data. Scribe's data-entry templates are defined in its database, with the possibility to assign different templates to different images or sets of images. As a result, we can use a simple template for a 1750 register of burials or a much more complex template for an 1881 census form. Since each transcribed record is linked to the section of the page image it represents, we have the ability to display the facsimile version of a record alongside its transcript in a list of search results, or to get fancy and pre-populate a transcriber's form with frequently-repeated information like months or birthplaces.

Under the guidance of Ben Laurie, the trustee directing the project, we are committed to open source and open data. We're releasing the source code under an Apache license and planning to build API access to the full set of record data.

We feel that the more the merrier in an open-source project, so we're looking for collaborators, whether they contribute code, funding, or advice. We are especially interested in collaborators from archives, libraries, and the genealogy world.

ReportersLab Reviews FromThePage

noreply@blogger.com (Ben W. Brumfield) — Wed, 03 Oct 2012 02:19:00 +0000

Tyler Dukes has written a concise introduction to the issues with handwritten material and a lovely review of FromThePage at ReportersLab:

Even when physical documents are converted into digital format, subtle inconsistencies in handwriting prove too much for optical character recognition software. The best computer scientists have been able to do is apply various machine learning techniques, but most of these require a lot of training data — accurate transcriptions deciphered by humans and fed into an algorithm.

“Fundamentally, I don’t think that we’re going to see effective OCR for freeform cursive any time soon,” Brumfield said. “The big successes so far with machine recognition have been in domains in which there’s a really constrained possibilities for what is written down.”

That means entries like numbers. Dates. Zip codes. Get beyond that, and you’re out of luck.

I don't know much about the world of investigative journalism, but it wouldn't surprise me if it holds as many intriguing parallels and new challenges as I've discovered among natural science collections. Handwriting might still be the most interdisciplinary technology.

Bilateral Digitization at Digital Frontiers 2012

noreply@blogger.com (Ben W. Brumfield) — Mon, 24 Sep 2012 19:01:00 +0000

This is a transcript of the talk I gave at Digital Frontiers 2012.

Abstract: One of the ironies of the Internet age is that traditional standards for accessibility have changed radically. Intelligent members of the public refer to undigitized manuscripts held in a research library as "locked away", even though anyone may study the well-cataloged, well-preserved material in the library's reading room. By the standard of 1992, institutionally-held manuscripts are far more accessible to researchers than uncatalogued materials in private collections -- especially when the term "private collections" includes over-stuffed suburban filing cabinets or unopened boxes inherited from the family archivist. In 2012, the democratization of digitization technology may favor informal collections over institutional ones, privileging online access over quality, completeness, preservation and professionalism.

Will the "cult of the amateur" destroy scholarly and archival standards? Will crowdsourcing unlock a vast, previously invisible archive of material scattered among the public for analysis by scholars? How can we influence the headlong rush to digitize through education and software design? This presentation will discuss the possibilities and challenges of mass digitization for amateurs, traditional scholars, libraries and archives, with a focus on handwritten documents.

My presentation is on bilateral digitzation: digitization done by institutions and by individuals outside of institutions and the wall that's sort of in between institutions and individuals.

In 1823, a young man named Jeremiah White Graves moved to Pittsylvania County, Virginia and started working as a clerk in a country store. Also that year he started recording a diary and journal of his experiences. He maintained this diary for the next fifty-five years, so it covers his experience -- his rise to become a relatively prominent landowner, tobacco farmer, and slaveholder. It covers the Civil War, it covers Reconstruction and the aftermath. (This is an entry covering Lee's surrender.)

In addition to the diary, he kept account books that give you details of plantation life that range from -- that you wouldn't otherwise see in the diaries. So for example, this is his daughter Fanny,

And this is a list of every single article of clothing that she took with her when she went off to a boarding school for a semester.

Perhaps more interesting, this is a memorandum of cash payments that he made to certain of his enslaved laborers for work on their customary holidays -- another sort of interesting factor. I got interested in this because I'm interested in the property that he lived in. The house that he built is now in my family, and I was doing some research on this. Since these account books include details of construction of the house, I spent a lot of time looking for these books. I've been looking for them for about the last ten years. I got in contact with some of the descendants of Jeremiah White Graves and found out through them that one of their ancestors had donated the diaries to the Alderman Library at the University of Virginia. I looked into getting them digitized and tried to get some collaboration [going] with some of the descendants, and one of them in particular, Alan Williams, was extremely helpful to me. But this was his reaction:

Okay. So we have diaries that are put in a library -- I believe one of the top research libraries in the country -- and they are behind a wall. They are locked away from him.

So let's talk about walls. From his perspective, the fact that these diaries--these family manuscripts of his--are in the Alderman Library means:

They're professionally conserved -- great!
They're publicly accessible, so anyone can walk in and look at them in the Reading Room.
They're cataloged, which would not be the case if they'd still been sitting in his family.
On the down side, they're a thousand miles away: they're in Virginia, he's in Florida, I'm in Texas. We all want to look at these, but it's awfully hard for people to get there if we don't have research budgets.
We have to deal with reading room restrictions if we actually get there.
Once we work on getting things digitized we have these permission-to-publish that we need to deal with, which have some moral challenges for someone from whose family these diaries came from.
And we have the scanning fees: the cost of getting them scanned by the excellent digitization department at the Alderman Library is a thousand dollars. Which is not unreasonable, but it's still pretty costly.

So here's a wall--a real, physical wall--between this institution and the public. How do we get through walls? Everyone here is familiar with digitization and collaboration. This is how we share things nowadays. It's how we've been sharing things for the last fifteen years, in fact. But, at least fifteen years ago, when we got started doing digitization, we had shallow digitization.

The prevalent practice in institutions was "scan-and-dump": make some scans, put them in a repository online.

One of the problems with that is that you have very limited metadata. The metadata is usually institutionally-oriented. No transcripts, in particular -- nobody has time for this. And quite often, they're in software platforms that are not crawlable by search engines.

Now meanwhile, amateurs are digitizing things, and they're doing something that's actually even worse! They are producing full transcripts, but they're not attaching them to any facsimiles. They're not including any provenance information or information about where their sources came from. Their editorial decisions about expanding abbreviations or any other sorts of modernizations or things like that -- they're invisible; none of those are documented.

Worst of all, however, is that the way that these things are propagated through the Internet is through cut-and-paste: so quite often from a website to a newsgroup to emails, you can't even find the original person who typed up whatever the source material was.

So how do we get to deep digitization and solve both of these problems?

The challenges to institutions, in my opinion, come down to funding and manpower. As we just mentioned, generally archives don't have a staff of people ready to produce documentary editions and put them online.

Outside institutions, the big challenge is standards; it is expertise. You've got manpower, you've got willingness, but you've got a lot of trouble making things work using the sorts of methodologies that have come out of the scholarly world and have been developed over the last hundred years.

So how do we fix these challenges?

One possible solution for institutions is crowdsourcing. We've talked about this this morning; we don't need to go into detail about what crowdsourcing is, but I'd like to talk a little bit about who participates in crowdsourcing projects and what kinds of things they can do and what this says about [crowdsourcing projects]. I've got three examples here. OldWeather.org is a project from GalaxyZoo, the Zooniverse/Citizen Science Alliance. The Zenas Matthews Diary was something that I collaborated with the Southwestern University Smith Library Special Collections on. And the Harry Ransom Center's Manuscript Fragments Project.

Okay, in Old Weather there are Royal Navy logbooks that record temperature measurements every four hours: the midshipman of the watch would come out on deck and record barometric pressure, wind speed, wind direction and temperature. This is of incredible importance for climate scientists because you cannot point a weather satellite at the south Pacific in 1916. The problem is that it's all handwritten and you need humans to transcribe this.

They launched this project three years ago, I believe, and they're done. They've transcribed all the Royal Navy logs from the period essentially around World War I -- all in triplicate. So blind triple keying every record. And the results are pretty impressive.

Each individual volunteer's transcripts tend to be about 97% accurate. For every thousand logbook entries, three entries are going to be wrong because of volunteer error. But this compares pretty favorably with the ten that are actually honestly illegible, or indeed the three that are the result of the midshipman of the watch confusing north and south.

So in terms of participation, OldWeather has gotten transcribed more than 1.6 million weather observations--again, all triple-keyed--through the efforts of sixteen thousand volunteers who've been transcribing pages from a million pages of logs.

So what this means is that you have a mean contribution of one hundred transcriptions per user. But that statistic is worthless!

Because you don't have individual volunteers transcribing one hundred things apiece. You don't have an even distribution. This is a color map of contributions per user. Each user has a square. The size of the square represents the quantity of records that they transcribed. And what you can see here is that of those 1.6 million records, fully a tenth (in the left-hand column) were transcribed by only ten users.

So we see this in other projects. This is a power-law distribution in which most of the contributions are made by a hand-full of "well-informed enthusiasts". I've talked elsewhere about how this is true in small projects as well. What I'd like to talk about here is some of the implications.

One of the implications is that very small projects can work: This is the Zenas Matthews Diaries that were transcribed on FromThePage by one single volunteer -- one well-informed enthusiast in fourteen days.

Before we had announced the project publicly he found it, transcribed the entire 43-page diary from the Mexican-American War of a Texas volunteer, went back and made two hundred and fifty revisions to those pages, and added two dozen footnotes.

This also has implications for the kinds of tasks you can ask volunteers to do. This is the Harry Ransom Center Manuscript Fragments Project in which the Ransom Center has a number of fragments of medieval manuscripts that were later used in binding for later works, and they're asking people to identify them so that perhaps they can reassemble them.

So here's a posting on Flickr. They're saying, "Please identify this in the comments thread."

And look: we've got people volunteering transcriptions of exactly what this is: identifying, "Hey, this is the Digest of Justinian, oh, and this is where you can go find this."

This is true even for smaller, more difficult fragments. Here we have one user going through and identifying just the left hand fragment of this chunk of manuscript that was used for binding.

So crowdsourcing and deep digitization has a virtuous cycle in my opinion. You go through and you try to engage volunteers to come do this kind of work. That generates deep digitization which means that these resources are findable. And because they're findable, you can find more volunteers.

I've had this happen recently with a personal project, transcribing my great-great grandmother's diary. The current top volunteer on this is a man named Nat Wooding. He's a retired data analyst from Halifax County, Virginia. He's transcribed a hundred pages and indexed them in six months. He has no relationship whatsoever to the diarist.

But his great uncle was the postman who's mentioned in the diaries, and once we had a few pages worth of transcripts done, he went online and did a vanity search for "Nat Wooding", found the postman--also named Nat Wooding--discovered that that was his great uncle and has become a volunteer.

Here's the example: this is just a scan/facsimile. Google can't read this.

Google can read this, and find Nat Wooding.

Now I'd like to turn to non-institutional digitization. I said "bilateral" -- this means, what happens when the public initiates digitization efforts. What are the challenges--I mentioned standards--how can we fix those. And why is this important?

Well, there is this--what I call the Invisible Archive, of privately held materials throughout the country and indeed the world. And most of it is not held by private collectors that are wealthy, like private art collectors. They are someone's great aunt who has things stashed away in filing cabinets in her basement. Or worse, they are the heirs of that great aunt, who aren't interested and have them stuck in boxes in their attic. We have primary sources here of non-notable subjects, that are very hard to study because you can't get at them.

But this is a problem that has been solved, outside of manuscripts. It's been solved with photographs. It's been solved by Flickr. Nowadays, if you want to find photographs of African-American girls from the 1960s on tricycles, you can find them on Flickr. Twenty years ago, this was something that was irretrievable. So Flickr is a good example, and I'd like to use it to describe how we might be able to apply it to other fields.

So, in terms of solving the standards problem, amateur digitization has a bad, bad reputation, as you can see here.

And much of that bad reputation is deserved, and isn't specific to digitization. This has been a problem with print editions in the past, it is a problem online now. Frankly, scholars don't trust the materials because they're not up to standard.

How do we solve this? Collaboration: we'd like to see more participation from people who are scholars, who are trained archivists, who are trained librarians to participate in some of these projects.

One of the ones I'm working with [is] digitizing these registers from the Reformation up to the present. We're building this generalizable, open-source, crowdsourced transcription tool and indexing tool for structured data. We'd love to find archivists to tell us what to do, what not to do, and to collaborate with us on this.

Another solution is community. You don't go on Flickr just to share your photos; you go on Flickr to learn to become a better photographer. And I think that creating platforms and creating communities that can come up with these standards and enforce them among themselves can really help.

The same thing is true with software platforms, if they actually prompt users and say: "when you're uploading this image, tell us about the provenance." "Maybe you might want to scan the frontispieces." "Maybe you'd like to tell us the history of ownership."

Those are the things that I think might get us there. I've just hit my time limit, I think, so thanks a lot!

Ben Brumfield is a family historian and independent software engineer. For the last seven years he has been developing FromThePage, an open source manuscript transcription tool in use by libraries, museums, and family historians. He is currently working with FreeUKGen to create an open source system for indexing images and transcribing structured, hand-written material. Contact Ben at benwbrum@gmail.com.

RMagick Lightning Talk at Austin On Rails

noreply@blogger.com (Ben W. Brumfield) — Wed, 25 Jul 2012 16:53:00 +0000

This is a transcript of my five-minute lightning talk at the July meeting of Austin On Rails, the local Ruby on Rails user group. It highlights a few things I've learned from my work on MyopicVicar and Autosplit.

Hi everyone, I'm Ben Brumfield and I've been a professional Ruby on Rails developer for three whole months, [applause] and I am here to talk about RMagick! RMagick is a wrapper for ImageMagick, which is great for processing images. These are the kinds of images that I process:

(I write software that helps people transcribe old handwriting.)

Now is this a pretty image? I don't know. We need to do some things with this, and this is how RMagick has helped me solve some of my problems. One of the things we need to do is to turn it into a thumbnail.

This is an example of RMagic used as a very simple wrapper for ImageMagick: All we're doing is creating an ImageList out of that file, we say image.thumbnail and pass it a scaling factor, and that will very quickly go through and transform that image. Write it to a file and we're done.

So that's kind of basic. RMagick does a lot of other basic kinds of things like this, like rotate or negate--sometimes I get files that are negatives--things like that.

What about the cool stuff? Here's an example of something that I have to deal with.

This is a parish registry from 1810, which would be fine, but this is a scan of a bad microfilm of a parish registry from 1810 that is tilted. Now my users need to draw rectangles around individual lines, and the tilt is going to really throw them off.

How badly is this tilted? If you're interested in seeing some RMagick code, here's a script that I wrote to draw a grid over that image or any other image: http://tinyurl.com/GridifyGist . And what you'll see is:

Man, it kind of sucks! You draw a rectangle over that and you're going to get all kinds of weird stuff.

Enter deskew. So here's an example of ruby code, very simple:

require 'RMagick' # Which is very idiosyncratically capital-R capital-M

s = Magick::ImageList.new('skew.jpg') # We get a new ImageList from skew.jpg

d = s.deskew # And poof!  Zap!

d.write('deskew.jpg') # We write out deskew and get this:

Now this looks kind of weird. The image looks off. But the image's contents are not off. And if we throw the grid on it:

You can see that what RMagick has done is that it's gone through and it's looked for lines inside the content of the image, and it's rotated the image in correspondence to where it thinks the lines go and where it thinks the orientation is.

I think that's pretty slick.

RMagick can do some other slick things, and this is where you get into the programming aspects of RMagick and why you'd use RMagick instead of just calling ImageMagick from the command line.

Here what we're doing is extracting files from a PDF. Lots of scanners produce images in PDF format, that is just using PDF as a container for a bunch of images.

This goes through, creates an ImageList from the file, and then goes through for each image in the image list--actually more specifically for each page in that PDF--and it writes it out correctly. (RMagick may not be the right thing to user for some files because it's going to get page by page images -- if you have PDFs that are composed of a bunch of images [on each page] then you're going to want to use some other tools.)

Okay. So here's a file that I'm dealing with, and when I want to present it to my users I actually want to present them with a single page. Is this file a single page? No -- this file is two pages, all scanned on a flatbed scanner. (For those that are curious, it's a sixteenth-century Spanish legal document; I don't know what it says.) But how do I find the spine? How do I know where to split it? It's easy enough to go through and do cropping, but you know, what do we do?

So I came up with this idea: Let's look for vertical dark stripes. Let's look for the darkest strip that is vertical in a deskewed version of this image and see if we can identify that. So this is something we can do. What I've done here is I've said-- this is the inside of a loop where I've said for each x let's pull all the pixels out, and then come up with a total brightness for that image [stripe]. Then later on, I'm going to find the minimum brightness for those vertical stripes.

If I do this on some of the files and indicate that stripe by a red line--which I hope you can see--it did pretty well on this!

It does well on this awful scan of a microfilm, although the red line is hard to see at this resolution.

And wow, it does great on this piece!

This is just an example of using RMagick to solve my problems. After I've gotten the line I want to crop it, so left page/right page

The only caveats that I'd give you about RMagick is that I find it necessary to call GC.start a lot -- at least I did in Ruby 1 6 [ed: 1.8.6] -- because RMagic--I don't know, man--because it swaps out the garbage collector or something and you run out of memory really fast.

RMagick: I love it!

Survey on Crowdsourced Transcription Tools

noreply@blogger.com (Ben W. Brumfield) — Thu, 31 May 2012 20:29:00 +0000

Via Twitter and the TEI-L mailing list, I see that Jens Brokfeld, a graduate student in Potsdam, is conducting a survey of projects using transcription tools for his thesis, “Creating Digital Editions with Crowdsourced Manuscript Transcription: A Tool Evaluation”. I encourage readers of this blog to take the survey and help advance the field.

In case that your organisation (archives, libraries, scientific institutions) works on projects for digital editions or may do so in the future, I would be very grateful for your answers to my survey questions. Please forward this e-mail to anyone working with digital editions and transcription tools. The survey will take 15-20 minutes.

Please click on the following link to complete the survey: https://www.surveymonkey.com/s/survey_transcription_tools

Transcription Tools at TCDL 2012

noreply@blogger.com (Ben W. Brumfield) — Fri, 25 May 2012 15:45:00 +0000

Yesterday I presented a guide to choosing software for crowdsourced manuscript transcription at the Texas Conference on Digital Libraries. Here are the slides from that talk:

Bending Regular Expressions to Express Uncertainty

noreply@blogger.com (Ben W. Brumfield) — Thu, 03 May 2012 02:24:00 +0000

This post expands upon an email I sent to James Edward Gray II, my mentor in regular expressions, whom I thank for his generosity.

For the last few months--ever since I started conducting regular expression workshops at THATCamps--I've been thinking about using regular expressions to represent uncertainty. The great power of the regex is its ability to define context, boundaries, and precision over what is essentially unknown: /A[BC]/ means "I'm looking for an A, followed by either a B or a C -- I don't know which, but if you see either of those, that's the text."

I build tools for transcribing handwritten text -- some of it very old, some of it illegible, some of it damaged by fire or water. Representing uncertainty is a common problem within that domain, and it's important for a couple of reasons:

Usually the text will be presented out of context -- the ASCII (or Unicode) characters representing the underlying text will be separated from the image containing the underlying text.
Even when the images are available, the person doing the transcription is far more skilled at deciphering a particular cursive hand than their readers are likely to be. Take someone trained in 16th century paleography, who has spent weeks working on a particular author's handwriting -- their opinion on whether a scribble is an "f" or an "s" is going to be worth more than that of a casual researcher who encounters that handwriting in a single record of search results. We need to pass that opinion from the expert to the reader.

While otherwise rigorous, the methods professional documentary editors use for recording uncertain readings are pretty shabby -- suited to print editions, they're often concise but imprecise: the Emerson Journals display missing text via || . . . ||, with "three dots representing one to five words; four dots, six to ten words; and five dots, sixteen to thirty words". Another example uses [ . . . ], with each period representing an illegible letter. There is no convention for expressing "I'm sure this is either an 'a' or a 'u', but I can't be certain which one." (Kline and Perdue, A Guide to Documentary Editing, Third Edition)

This is where the notation regular expressions use for their search patterns comes in. It seems like a perfect fit to record Br[au]mfield when the user can't tell whether a name is "Brumfield" or "Bramfield" but is certain that it's not, say "Bremfield". And happily, FreeUKGen is doing exactly this -- they've created a regex-inspired Uncertain Character Format (UCF) for their volunteers to use when they're not quite able to make out text:

_ (Underscore) A single uncertain character. It could be anything but is definitely one character. It can be repeated for each uncertain character.

* (Asterisk) Several adjacent uncertain characters. A single * is used when there are 1 or more adjacent uncertain characters. It is not used immediately before or after a _ or another *.
Note: If it is clear there is a space, then * * is used to represent 2 words, neither of which can be read.

[abc] A single character that could be any one of the contained characters and only those characters. There must be at least two characters between the brackets.
For example, [79] would mean either a 7 or a 9, whereas [C_] would mean a C or some other character.

{min,max} Repeat count - the preceding character occurs somewhere between min and max times. max may be omitted, meaning there is no upper limit. So _{1,} would be equivalent to *, and _{0,1} means that it is unclear if there is any character.

? Sometimes you will have the situation where all of the characters have been read but you remain uncertain of the word. In this case append a ? at the end of the word e.g. RACHARD? The most frequent place where a ? is used is with transcription that have been donated from other systems and are being converted for entry into FreeREG.

And their volunteers are actually using UCF a lot -- here's a file with a minimal-but-effective example, and here's a list of files with a high incidence of records containing UCF.

One of the only big differences between UCF notation and regular expression notation is the use of underscore (_) for "I'm not sure what this character is". In a sense, this is equivalent to the regex . character, but in practice that's not how it's used. /[i.]/ makes no sense in regular expressions: "either 'i' or any character" is turned into the logical set of all characters plus the set of { 'i' }, the union of which is the same as the set of all characters. As a result, in regular expressions the 'i' is redundant in [i.]. However, that's not how they use _ in UCF. [i_] means "I think this character is an 'i', but I'm not really sure." That statement is not the same thing as "I don't know what this character is" -- not at all!

So, cool! We've got a notation for describing uncertainty inspired by regular expressions. Problem solved, right? Well, not quite. While FreeUKGen's UCF represents uncertain readings successfully, I think, there are still a couple of issues to iron out.

The first one of these is displaying the data -- I won't go too far into this, as UI is not really my forte, but it seems like we might be able to represent notations of the form "[a_]" by using a different font weight. I have no idea what we'll do about a "Br[au]mfield", though.

The second issue is in searching the data. No problem, right? Regular expressions are designed for searching! Well, sort of. In this case, we expect end users (primarily genealogy researchers) to be typing in precise search strings like "Brumfield" which they expect to match against the regular expression /Br[au]mfield/. This wouldn't be a problem if we only had a handful of records -- we'd convert the UCF in each transcription into its equivalent regular expression, then iterate through each record, matching it against the user-entered search string. Unfortunately this approach might take a while on a database containing hundreds of millions of records.

The problem with searching regular expressions is that you can't index them. So far as I'm aware, it's theoretically impossible to shove a working finite state machine into a B+-tree. What you can do, however, is index permutations of UCF -- if a first name is [_J]ane, you can at least index Jane so that a search for "Jane" will find that record. You can also permute /Br[au]mfield/ into both "Brumfield" and "Bramfield" and index them each, so that a search on either string will find the /Br[au]mfield/ record. This an incomplete solution, in that its results will differ from the aforementioned, logically correct approach of applying each regex against the search string. However, it might be just adequate for the most common cases.

After writing this and reading James's response, I started thinking more about my options. One of these is a parallelized brute-force approach. Why can't I match each regex in the database against a search string? After all, we're talking about fewer than a billion records, and asking does X match Y is the sort of thing that is easily parallelized. O brave new world, that has such infrastructure! I'm hesitant to go down this path, but I may be missing something -- perhaps some Hadoopy, Erlangy, Map-Reducey algorithm is cheap, easy, and presents the simplest solution to the problem? Any other option is really an approximation to "correct", so it would be a shame to rule this out because of my own lack of experience.

Another approach might be to categorize each kind of UCF expression. Based on my limited research so far, it appears that the majority of the UCF in the existing transcripts falls into either the "completely unknown" category of "*" for an entire field, or the nuanced "I think this is a J but I'm not sure" category represented by [J_]. We will likely have to handle the former gingerly no matter what we do -- if a surname is totally illegible in the manuscript, the search engine will have to rely on other fields. The former expression could be approximated by "J", which would match precise searches well provided the transcriber actually has the greatest possible expertise.

Moving into what I expect are rarer cases, expressions like /Br[au]mfield/ would work well with the permutation treatment I outlined above. If the system already supports begins-with and ends-with searches, we should be able to index "Brumf*" as well as "*field". In fact, we might even be able to index a single, infixed wildcard like "Br*ld" by doing both begins-with and ends-with searches with a combination of cleverness and hackery. This leaves some smaller number of true regular expression-equivalent UCF-encoded records like "Ca_{1,2}s*d" to deal with. It's possible that this represents such a small sample that the system could actually apply each record containing an irreducible regex to every search, whether via big parallelization or a long loop.

Yet another approach is one that I understand to be deployed on the FreeUKGen databases now -- lossy comparison that reduces UCF to searchable data. For example, the venerable Soundex algorithm begins by stripping non-alphabetic data from a record, converting /[J_]ane/ to "Jane". The uncertainty recorded by the transcriber fades into the larger fog of the search algorithm. I'm just as uncomfortable with this methodology as I am with the permutation of /[J_]ane/ to "Jane" I described above. I suspect that my discomfort is due to simply not knowing what the correct behavior is when a user is searching on /[J_]ane/ -- I know that "Jane" should match, but am not entirely sure whether "Zane" should match the record.

Perhaps the right approach is a hybrid -- use tricks with database indexing for the majority of cases--which don't involve any UCF at all--provide Soundex and Metaphone in transparent ways, and shove the irreducable regular expressions into a spot where they can be processed cheaply. But I really don't know, and I don't anticipate knowing for months yet. Of course, if you happen to have done this before, I'd love to know how. I'm heading to bed, expecting dreams which revolve around gem install index-fsm.

Crowdsourced Transcription Tool List

noreply@blogger.com (Ben W. Brumfield) — Wed, 11 Apr 2012 10:34:00 +0000

When I first started this blog, I spent a lot of time writing detailed reviews of different transcription projects. This has become difficult as my available time shrinks and the number of crowdsourcing projects grows. So when Kate Bowers posted to the Society of American Archivists mailing list asking for a directory of transcription tools, I figured it was time to take a different approach.

http://tinyurl.com/TranscriptionToolGDoc

The link above is a Google Documents spreadsheet listing different tools and the features I thought were relevant. It's been updated several times over the last few weeks, and I'm pleased to see that it's expanded to include a score of technologies. I hope it's useful.

French Departmental Archive on Wikisource

noreply@blogger.com (Ben W. Brumfield) — Wed, 04 Apr 2012 17:47:00 +0000

While the transcription world buzzes with news of the release of the 1940 US census and the crowdsourced transcription projects that surround it, I'd like to draw your attention to a blog post published last week on La Tribune des Archives: "Edition collaborative de manuscrits sur Wikisource : 1er retour d'expérience". The post covers the efforts of the archives of the department of Alpes-Maritimes to transcribe 17th- and 18th-century records of episcopal visits to the communes in the diocese. These records are rich sources on local history, but "readers struggle over the chicken-scratch, and the collection is too large to be edited by a single person." The archive has used Wikisource.fr to transcribe these manuscripts with great success, so I'd like to quote and translate extensive portions of their post.

Why Wikisource?
It's already there! (No software to create, maintain, administer, no specs -- just a strong will and a a core of 2-6 people).
It offers features designed for manuscript editions requiring more than one editor.
Particularly useful functions (aside from the collaborative aspect) :

Side-by-side display of facsimile and transcription

Workflow indicating whether a page is transcribed, corrected, or validated by two administrators.

The visualization is very practical for motivating the community of transcribers.

Version history control and the ability to comment or discuss difficult issues.

Wikisource's high Google page rank.

The article continues to describe the factors they weighed when choosing material for the project (accessibility of the script and local interest, among others), how they got started (the standard GLAMWiki approach), then continues to the community management aspects I find so fascinating:

How do you motivate your paleographers?
In our experience, transcribers are essentially former university students and internally-trained archivists who want to extend their education (either by making further progress or by avoiding becoming rusty).
Work times and rest times clearly defined in advance.
A regular, fixed-date schedule defined in advance (for example, one month: upload on the 15th and correction every last day of the month) helps the group to make progess and to break up its efforts with relaxation periods (for the eyes, the editors, and the correctors) and lets everyone have rapid feedback (new pages are in fact corrected practically every night).

Findings on the behavior of "students" on Wikisource
The first exercises attracted the kind help support of Wikisource regulars and administrators (Adrienne Alix, SereinWMfr, Pyb, Hsarrazin), a few new registered paleographers (Cavalié, LINCK, Braxmeyer, Gustave) and some anonymous IPs. One or two correctors can suffice easily to keep track of the work of 5-10 "students". Contrary to homework done in class, the "students" apply themselves regularly to the task, and the size and number of contributors does not increase on the night before the deadline.
Writings dating from before 1660 receive fewer volunteers but could very well serve as university exercises graded online (at the rate of one page per student).

For more on the archive's efforts (including their similar outreach on Flickr), take a look at the departmental archive news page.

Crowdsourcing at IMLS WebWise 2012

noreply@blogger.com (Ben W. Brumfield) — Sat, 17 Mar 2012 22:15:00 +0000

The video of the crowdsourcing panel at IMLS WebWise is online, so I thought I'd post my talk. Like anyone who's created a transcript of their own unscripted remarks, I recommend watching the video. (My bit starts at 6:00, though all the speakers were excellent. Full-screening will hide the slides.) Nevertheless, I've added hyperlinks to the transcript and interpolated the slides with my comments below.

Okay. I'd like to talk about some of the lessons that have come out of my collaborations with small crowdsourcing projects. We hear a lot about these large projects like GalaxyZoo, like Transcribe Bentham. What can small institutions and small projects do, and do the rules that seem to apply to large projects also apply to them?

So there are three projects that I'm drawing from here in this experience. The first one I'm going to talk about in a little bit is one that was run by Balboa Park Online Collaborative. It's a Klauber Field Notes Transcription Project--the field notes of Laurence M. Klauber, who was the nation's foremost authority on rattlesnakes. These are field notes that he kept from 1923 through 1967. This is done by the San Diego Natural History Museum and is run by our own Perian Sully who is out there in the room somewhere.

The next project I want to talk about is the Diary of Zenas Matthews. Zenas Matthews was a volunteer from Texas who served in the American forces in the US-Mexican War of 1846 and this diary is kept by Southwestern's article on the Zenas Matthews diary project. This had been digitized for a previous researcher and is small, but Southwestern itself is also quite small.

The third project I want to talk about is actually the origin of the software, which is the Julia Brumfield Diaries. If the name looks familiar, it's because she's my great-great grandmother. This project was the impetus for me to develop this tool for crowdsourced transcription.

So all of these projects, what they have in common is that we're talking about page counts that are in the thousands and volunteer counts that are numbered in the dozens at best. So these are not FamilySearch Indexing, where you can rely on hundreds of thousands of volunteers and large networks.

So who participates in large projects and who participates in small projects? One thing that I think is really interesting about crowdsourcing and these other sorts of participatory online communities is that the ratio of contributions to users follows what's called a power-law distribution.

If you look here, we see--and most famously this is Wikipedia--and you see a chart of the number of users on Wikipedia ranked by their contributions. And what you see is that 90% of the edits made to Wikipedia are done by 10% of the users.

If we look at other crowdsourced projects: this is the North American Bird Phenology Program out of Patuxent ~~Bay~~ Research Center [ed: actually Patuxent Wildlife Research Center], and this is a project in which volunteers are transcribing ornithology records--basically bird-watching records--that were sent in from the ~~1870s through the 1950s~~ [ed: 1880s-1970s], entering them into a database where they can be mined for climate change [data]. What's interesting about this to me at least is that--and this has been a phenomenally successful project: they've got 560,000 cards transcribed all by volunteers, but StellaW@Maine here has transcribed 126,000 of them, which is 22% of them. Now, CharlotteC@Maryland is close behind her (so go, local team!) but again you see the same kind of curve.

If we look at another relatively large project, the Transcribe Bentham project: this isn't a graph, but if you look at the numbers here, you see the same kind of thing. You see Diane with 78,000 points, you see Ben Pokowski with 51,000 points. You see this curve sort of taper down into more of a long tail.

So what about the small projects?

Well, let's look at the Klauber diaries. This is the top ten transcribers of the field notes of Laurence Klauber. And if you look at the numbers here--again in this case it's not quite as pronounced because I think the previous leader has dropped out and other people have overtaken him--but you see the same kind of distribution. This is not a linear progression; this is more of a power-law distribution.

If you look at an even smaller project--now, mind you this is a project that is really only of interest to members of my family and elderly neighbors of the diarist--but look: We've got Linda Tucker who has transcribed 713 of these pages followed by me and a few other people. But again, you have this power law that the majority of the work is being done by a very small group of people.

Okay, what's going on really? What does this mean and why does it matter? The thing that I think this gets to, the reason that I think that this is important, is for a couple of reasons.

One is that this kind of behavior addresses one of the main objections to crowdsourcing. Now there are a lot of valid objections to crowdsourcing; I think that there are also a few invalid objections and one of them is essentially the idea that members of the public cannot participate in scholarly projects because my next door neighbor is neither capable nor interested in participating in scholarly projects. And we see this all over the place. I mean, here's a few example quotes--and I'm not going to read them out. I believe that this objection (which I have heard a number of times; I mean we see some examples right here) is a non sequitur. And I believe that the power-law distribution proves that it's a non sequitur. Really, I saw this most egregiously framed by a scholar who was passionately--just absolutely decrying--the idea that classical music fans would be able to competently translate from German into English because, he said, "After all, 40% of South Carolina voted for Newt Gingrich." Okay.

All right, so what's going on is I think best summed up by Rachel Stone, and what she essentially said is that crowdsourcing isn't getting the sort of random distribution from the crowd. Crowdsourcing is getting a number of "well-informed enthusiasts."

So where do we find well-informed enthusiasts to do this work and to do it well? Big projects have an advantage, right? They have marketing budgets. They have press coverage. They have an existing user base.

If you ask the people at the Transcrbe Bentham project how did they get their users, they'll say "Well, you know that New York Times article really helped. " That's cool! All right.

The GalaxyZoo people--Citizen Science Alliance--yesterday, 24 hours ago, announced a new project, SETILive. Now what this does is it pulls in live data from the SETI satellites[sic: actually telescope], and in those 24 hours--I took this screenshot; I actually skipped lunch to get this one screenshot because I knew that it would pass 10,000 people participating with 80,000 of these classifications. And it would have been higher, except last night the telescope got covered by cloud cover. So they dropped from getting 30 to 40 contributions per second to having to show sort of archival data and getting only 10 contributions per second. Well, they can do this because they have an existing base of active volunteers that numbers around 600,000.

So how do WE do that? How do we find well-informed enthusiasts? This is something that Kathryn Stallard and Anne Veerkamp-Andersen at Southwestern University Special Collections and I discussed a lot when we were trying to launch the Zenas Matthews Diary. We said, "Well, we don't have any budget at all." Kathryn said, "Well, let's talk about local archival newsletters. Let's post to H-Net lists." I was in favor of looking at online communities of people who might be doing Matthews genealogy or the military history war-gamers who have discussion forums on the Mexican War.

While we're arguing about this, Kathryn gets an email from a patron saying, "Hey, I'm a member of an organization. We see that you have this document. It relates to the Battle of San Jacinto and the Texas Revolution of 1836. Can you send this to us?"

She responds saying, "Hey, 1846, great. Check out this diary we just put online. I think that's what you're talking about."

Well that wasn't actually what he was talking about, but he responds and says, "Yeah, okay, I'll check that out, but can you please give me the document I want." They get it back to him and we returned to our discussion of "Okay, what do we need to do to roll this out? We're going to start working on the information architecture. We're going to work on the UI. We're going to work on help screens." And while we're having this conversation, Mr. Patrick checks it out.

And Scott Patrick starts transcribing.

And he starts transcribing some more.

And he continues transcribing.

And at this point, we're talking about working on the wording of the help screens, the wording of our announcement trying to attract volunteers, and this is page 43 of the 43-page diary!

And while we're discussing this, he goes back and he starts adding footnotes. Look at this: he's identifying the people who are in this, saying, "Hey, this guy who is mentioned is -- here's what his later life was. This other guy--hey, he's my first cousin, by the way, but he also left the governorship of the State of Texas to fight in this war."

He sees--and believe me, in the actual original diary, Piloncillo is not spelled Piloncillo. I mean it is a -- Zenas Matthews does not know Spanish, right? He identifies this! He identifies and looks up works that are mentioned here.

So wow! All right! We got our well-informed enthusiast! In 14 days, he transcribed the diary, and he didn't do just one pass. I mean as he got familiar with the hand, he goes back and revises the earlier transcriptions. He kind of figures out who's involved. He asks other members of his heritage organization what this is. He adds two dozen footnotes.

What just happened? What was that about? Who is this guy? Well, Scott Patrick is a retired petroleum worker who got interested in his family history, and then got interested in local history, and then got interested in heritage organizations. And he is our ideal "well-informed enthusiast".

So how did we find him? The project isn't public yet, right? Our challenge now is rephrasing our public announcement. We're now looking for volunteers to ... something that adequately describes what's left to do. Well, let's go back and take a look at this original letter, right? What we did is, we responded to an inquiry from a patron--and not an in-person patron: this is someone who lives 200 miles away from Georgetown, Texas.

What you have when someone is coming in and asking about material is, if you think about this in terms of target marketing--this is a target-rich environment. Here is someone who is interested. He's online. He's researching this particular subject. He is not an existing patron. he has no prior relationship with Southwestern University Libraries, but "Hey, while we answer your request, you might check this thing out that's in this related field." That seems to have worked in this one case. Hopefully, we'll get some more experience with future projects.

Okay, so how do we motivate volunteers? More importantly, how do we avoid de-motivating them?

Big projects, a lot of times they have a lot of interesting game-like features. Some of them actually are games. You have leader boards, you have badges, you have ways of making the experience more immersive.

OldWeather, which is run by GalaxyZoo, will plot your ship on a Google map as you transcribe the latitude and longitude elements from the log books.

The National Library of Finland has partnered with Microtask to actually create a crowdsourcing game of Whac-A-Mole. So this is crowdsourcing taken to the extreme.

But there's a peril here, and the peril is that all of these things are extrinsic motivators.

And we ran into this with the Klauber diaries. Perian came to me and said, "Hey, let's come up with a stats page, because we want to track where the diaries are at. So we come up with the stats page -- pretty basic, here's where some of these are at.

And hey, while we're at it, let's mine our data. We can come up with a couple of top-10 lists. So we come up with the top-ten list of transcribers and a top-ten list of editors, because that's the data I have.

Well remember, the whole point of this exercise is to index these diaries so that we can find the mentions of these individual species in the original manuscripts. Do you see indexing on here anywhere? Neither did our volunteers, and the minute this went up, the volunteers who previously had been transcribing and indexing every single page stopped indexing completely. They weren't being measured on it. We weren't saying that we rewarded them for it, so they stopped.

Needless to say, our next big-rush change was a top-ten indexers.

So this gets to "crowding-out" theory of motivation, and the expert on this is a researcher in the UK named Alexandra Eveleigh. Her point is that if you're going to design any kind of extrinsic motivation, you have to make sure that it promotes the actual contributory behavior, and this is something that applies, I believe, to small projects as well as large projects.

So I have 13 seconds left, so thank you, and I'll just end on that note.

Jumping In With Both Feet

noreply@blogger.com (Ben W. Brumfield) — Thu, 08 Mar 2012 16:03:00 +0000

Although I didn't know it at the time, since I began work on FromThePage in 2005 I've had one toe in the digital humanities community. I've worked on FromThePage and I've blogged about crowdsourced manuscript transcription. I've met some smart, friendly people doing fascinating things and I've even taught some of them the magic of regular expressions. But I've always tried to squeeze this work into my "spare time" -- the interstices in the daily life of an involved father and a professional software engineer working a demanding but rewarding job. As the demands of vocation and avocation increase; as disparate duties begin to compete with each other; as new babies come into my home while new technologies come into my workplace and new requests for FromThePage arrive in my inbox, the basement inventor model becomes increasingly untenable. The numbers don't lie: I've only checked in code on four days during the last six months.

In January I was offered an incredible opportunity. Chris Lintott invited me to the Adler Planetarium to meet the Citizen Science Alliance's dev team. This talented, generous team of astronomer-developers gave me a behind-the-scenes tour of their Scribe tool--early versions of which powered OldWeather.org--and I was blown away. I don't think I've ever been so excited about a technology, and my mind raced with ideas for projects using it. . Serendipitously, two days later I received email from Ben Laurie asking if I'd like to implement Scribe for the FreeREG project, a part of the FreeBMD genealogy charity that is transcribing parish registers recording the baptisms, marriages, and burials in England and Wales from 1538-1835. All development would be released open source, and all data would be as open as possible. It's a dream project for someone with my interests; there was no way I could pass this up.

So as of March 18 I'm starting a new career as an independent digital history developer. It is heartbreaking to leave my friends at Convio after nearly a dozen years, but I'm delighted with the possibilities my new autonomy offers. I hope to specialize in projects relating to crowdsourcing and/or manuscript transcription, but to be honest I'm not sure where this path will lead. Of course I plan to devote more time to FromThePage -- this year should finally see the publish-on-demand integration I've always been wishing for, as well as a few other features people have requested. If you've got a project that seems appropriate--whether it involves genealogy or herpetology, agricultural history or textile history--drop me a line.

Quality Control for Crowdsourced Transcription

noreply@blogger.com (Ben W. Brumfield) — Mon, 05 Mar 2012 15:21:00 +0000

Whenever I talk about crowd-sourced transcription--actually whenever I talk about crowdsourced anything--the first question people ask is about accuracy. Nobody trusts the public add to an institution's data/meta-data, nor especially to correct it. However, quality control over data entry is a well-explored problem, and while I'm not familiar with the literature from industry regarding commercial approaches, I'd like to offer the systems I've seen implemented in the kinds of volunteer transcription projects I follow. (Note: the terminology is my own, and may be non-standard.)

Single-track methods (mainly employed with large, prosy text that is difficult to compare against independent transcriptions of the same text). In these methods, all changes and corrections are made to a single transcription which originated with a volunteer and is modified thereafter. There no parallel/alternate transcription to compare against.
1. Open-ended community revision: This is the method that Wikipedia uses, and it's the strategy I've followed in FromThePage. In this method, users may continue to change the text of a transcription forever. Because all changes are logged--with a pointer of some sort to the user who logged them--vandalism or edits which are not in good faith may be reverted to a known-good state easily. This is in keeping with the digital humanities principle of "no final version." In my own projects, I've seen edits made to a transcription two decades after the initial version, and those changes were indeed correct. (Who knew that "drugget" was a coarse fabric used for covering tobacco plant-beds?) Furthermore, I believe that there is no reason other than the cost of implementation why any of the methods below which operate from the "final version" mind-set should not allow error reports against their "published" form.
2. Fixed-term community revision: Early versions of both TransribeBentham and Scripto followed this model, and while I'm not sure if either of them still do, it does seem to appeal to traditional documentary editing projects that are incorporating crowdsourcing as a valuable initial input to a project while wishing to retain ultimate control over the "final version". In this model, wiki-like systems are used to gather the inital data, with periodic review by experts. Once a transcription reaches an acceptable status (deemed so by the experts), it is locked to further community edits and the transcription is "published" to a more traditional medium like a CMS or a print edition.
3. Community-controlled revision work-flows: This model is a cross between the two above-mentioned methods. Like fixed-term revision, it embraces the concept of a "final version", after which the text may not be modified. Unlike fixed-term revision, there are no experts involved here -- rather the tool itself forces a text to go through an edit/review/proofread/reject-approve workflow by the community, after which the version is locked for future edits. As far as I'm aware, this is only implemented by the ProofreadPage plugin to MediaWiki that has been used by Wikisource for the past few years, but it seems quite effective.
4. Transcription with "known-bad" insertions before proofreading: This is a two-phase process, which to my knowledge has only been tried by the Written Rummage project as described in Code4Lib issue 15. In the first phase, an initial transcription is solicited from the crowd (which in their case is a Mechanical Turk workforce willing to transcribe 19th-century diaries for around eight cents per page). In the second phase, the crowd is asked to review the initial transcription against the original image, proof-reading and correcting the first transcription. In order to make sure that a review is effective, however, extra words/characters are added to the data before it is presented to the proof-reader, and the location within the text of these known-bad insertions is recorded. The resulting corrected transcription is then programmatially searched for the bad data which had been inserted, and if it has been removed the system assumes that any other errors have also been removed -- or at least that a good-faith effort has been made to proofread and correct the transcript.
5. Single-keying with expert review: In this methodology, once a single volunteer contribution is made, it is reviewed by an expert and either approved or rejected. The expert is necessarily authorized in some sense -- in the case of the UIowa Civil War Diaries, the review is done by the library staff member processing the mailto form contribution, while in the case of FreeREG the expert is a "syndicate manager" -- a particular kind of volunteer within the FreeBMD charity. (FreeREG may be unique in using a single-track method for small, structured records, however it demands more paleographic and linguistic expertise from its volunteers than any other project I'm aware of.) If a transcription is rejected, it may be either returned to the submitter for correction or corrected by the expert and published in corrected form.
Multi-track methods (mainly employed with easily-isolated, structured records like census entries or ship's log books). In all of these cases, the same image is presented to different users to be transcribed from scratch. The data thus collected is compared programmatically on the assumption that two correct transcriptions will agree with each other and may be assumed to be valid. If the two transcriptions disagree with each other, however, one of them must be in error, so some kind of programmatic or human expert intervention is needed. It should be noted that all of these methodologys are technically "blind" n-way keying, as the volunteers are unaware of each other's contributions and do not know whether they are interpreting the data for the first time or contributing a duplicate entry.
1. Triple-keying with voting: This is the method that the Zooniverse OldWeather team uses. Originally the OldWeather team collected the same information in ten different independent tracks, entered by users who were unaware of each other's contributions: blind, ten-way keying. The assumption was that majority reading would be the correct one, so essentially this is a voting system. After some analysis it was determined that the quality of three-way keying was indistinguishable from that of ten-way keying, so the system was modified to a less-skeptical algorithm, saving volunteer effort. If I understand correctly, the same kind of voting methodology is used by ReCAPTCHA for its OCR correction, which allowed its exploitation by 4chan.
2. Double-keying with expert reconciliation: In this system, the same entry is shown to two different volunteers, and if their submissions do not agree it is passed to an expert for reconciliation. This requires a second level of correction software capable of displaying the original image along with both submitted transcriptions. If I recall my fellow panelist David Klevan's WebWise presentation correctly, this system is used by the Holocaust Museum for one of their crowdsourcing projects.
3. Double-keying with emergent community-expert reconciliation: This method is almost identical to the previous one, with one important exception. The experts who reconcile divergent transcriptions are themselves volunteers -- volunteers who have been promoted to from transcribers to reconcilers through an algorithm. If a user has submitted a certain (large) number of transcriptions, and if those transcriptions have either 1) matched their counterpart's submission, or 2) been deemed correct by the reconciler when they are in conflict with their counterpart's transcription, then the user is automatically promoted. After promotion, they are able to choose their volunteer activity from either the queue of images to be transcribed or the queue of conflicting transcriptions to be reconciled. This is the system used by FamilySearch Indexing, and its emergent nature makes it a particularly scalable solution for quality control.
4. Double-keying with N-keyed run-off votes: Nobody actually does this that I'm aware of, but I think it might be cost-effective. If the initial set of two volunteer submissions don't agree, rather than submit the argument to an expert, re-queue the transcription to new volunteers. I'm not sure what the right number is here -- perhaps only a single tie-breaker vote, but perhaps three new volunteers to provide an overwhelming consensus against the original readings. If this is indecisive, why not re-submit the transcription again to an even larger group? Obviously this requires some limits, or else the whole thing could spiral into an infinite loop in which your entire pool of volunteers are arguing with each other about the reading of a single entry that is truly indecipherable. However, I think it has some promise as it may have the same scalability benefits of the previous method without needing the complex promotion algorithm nor the reconciliation UI.

Caveats: Some things are simply not knowable. It is hard to evaluate the effectiveness of quality control seriously without taking into account the possibility that volunteer contributors may be correct and experts may be wrong, nor more importantly that some images are simply illegible regardless of the paleographic expertise of the transcriber. The Zooniverse team is now exploring ways for volunteers to correct errors made not by transcribers but rather by the midshipmen of the watch who recorded the original entries a century ago. They realize that a mistaken "E" for "W" in a longitude record may be more amenable to correction than a truly illegible entry. Not all errors are made by the "crowd", after all.

Much of this list is based on observation of working sites and extrapolation, rather than any inside information. I welcome corrections and additions in the comments or at benwbrum@gmail.com.

[Update 2012-03-07: Folks from the Transcribe Bentham informed me on Twitter that "In general, at the moment most transcripts are worked on by one volunteer, checked and then locked. Vols seem to prefer working on fresh MSS to part transcribed." and "For the record, @TranscriBentham does still use 'Fixed-term community revision'. There are weekly updates on the blog." Thanks, Tim and Justin!]

We Get Press!

noreply@blogger.com (Ben W. Brumfield) — Sun, 04 Mar 2012 22:30:00 +0000

Crowdsourced transcription projects--and FromThePage in particular--have gotten some really nice press in the last few weeks.

Konrad Lawson posted an excellent review of Scripto and FromThePage on the ProfHacker blog at The Chronicle of Higher Education: Crowdsourcing Transcription: FromThePage and Scripto.

Francine Diep wrote a great article on the phenomenon at Innovation News Daily: Volunteer Transcribers Put Millions of Pages Online.

Ellen Davis's article on Southwestern's transcription of Zenas Matthews's 1846 Mexican War Diary is especially notable because it includes an interview with Scott Patrick, the volunteer who has done such a spectacular job: Collaborative Transcription Project.

`_` (Underscore)	A single uncertain character. It could be anything but is definitely one character. It can be repeated for each uncertain character.
`*` (Asterisk)	Several adjacent uncertain characters. A single `` is used when there are 1 or more adjacent uncertain characters. It is not used immediately before or after a `_` or another ``. Note: If it is clear there is a space, then `* *` is used to represent 2 words, neither of which can be read.
`[abc]`	A single character that could be any one of the contained characters and only those characters. There must be at least two characters between the brackets. For example, `[79]` would mean either a `7` or a `9`, whereas `[C_]` would mean a `C` or some other character.
`{min,max}`	Repeat count - the preceding character occurs somewhere between min and max times. max may be omitted, meaning there is no upper limit. So `_{1,}` would be equivalent to `*`, and `_{0,1}` means that it is unclear if there is any character.
`?`	Sometimes you will have the situation where all of the characters have been read but you remain uncertain of the word. In this case append a ? at the end of the word e.g. RACHARD? The most frequent place where a ? is used is with transcription that have been donated from other systems and are being converted for entry into FreeREG.