I plan to leave this web site up for a few more weeks and then hand the .org name over to Patrick. During that time, some of the content from this site will be merged into the framework of his new site. Please go over to the new site and participate in the forums.

I plan continue blogging about reproducible research from time to time, but future posts will be on my personal blog, The Endeavour. I may write a few more posts here regarding the status of the transition.

Long Now is trying to preserve documentation for thousands of years, but I just want to know how to preserve documents even for a few months or years. They want to hold on to knowledge as civilizations come and go. I’m just trying to hold on to knowledge as personnel come and go.

Mundane document preservation is a very difficult problem. Preserving the Declaration of Independence is easy; preserving meeting notes is hard. Preserving the Declaration is a technical problem. If you keep it in a glass case filled with nitrogen, keep the lights low, and make sure Nicolas Cage doesn’t steal it, you’re OK. Millions of people know that the document exists, and they know where to look for it. And besides the original paper copy, the text is available electronically in countless locations.

How do I preserve the document that describes why my internal software application uses the parameters it does? Make notes in the source code? Good idea, but most of the people who want to know about the parameters are not software developers. What about version control systems or content management systems? Great idea: put everything associated with a project in one place. But wherever you put the information, someone has to remember that it exists and know where to look for it.

The tagline for the Orfeo Toolbox project is “Orfeo Toolbox is not a black box.”

The project is located at  http://conceptual.sourceforge.net/ and is described in the paper Reproducible network benchmarks with coNCePTuaL by Scott Pakin of Los Alamos National Laboratory.

Some of the highlights of coNCePTuaL:

• Performance tests (timed network-communication patterns) are described in a precise but English-like “executable pseudocode” designed for basic readability, even by someone not familiar with coNCePTuaL.
• Output files produced by coNCePTuaL-based performance tests include not only the measurements but the code describing the  test itself and a detailed description of the experimental platform on which the code ran.  This enables a third party to see exactly what was run, how it was run, and what the results were, all in one file.
• coNCePTuaL can automatically produce space-time diagrams of the communication pattern for additional clarity of presentation.

Sweave has the potential to make statistical analyses more reproducible. But I doubt many realize its vulnerabilities. The Sweave files are likely to have implicit dependencies on R session state or data located outside the file. You don’t really know that the output is reproducible until it’s compiled by someone else in a fresh environment.

My proposal is a service that lets you submit an Sweave file and get back the resulting LaTeX and PDF output. An extension to this would allow users to also upload data files along with their Sweave file so not all data would have to be in the Sweave file itself. For good measure, there should be some checksums to certify just what input went into producing the output.

Here’s one way I see this being used. Suppose you’re about to put a project on the shelf for a while. For example, you’re about to submit a paper to a journal.You may need to come back and make changes six months later. You think about the difficulty you’ve had in the past with these sorts of edits and want to make sure it doesn’t happen again. So you submit your Sweave document to the build server to verify that it is self-contained.

Here’s another scenario. Suppose you’ve asked someone whom you supervise to produce a report. Instead of letting them give you a PDF, you might insist they give you an Sweave file that you then run through the build service to make your own PDF. That way you can have the whole “but it works on my machine” discussion now rather than having it months later after the person who make the report has a new computer or a new job.

In this talk, we will describe how we have analyzed the data, and the implications of the ambiguities for the clinical findings. We will also describe methods for making such analyses more reproducible, so that progress can be made more steadily.

The talk will be at 4 PM in Keck Hall room 102.

Even when your source code isn’t changing, the environment around it is changing. When I was in college, computers didn’t have automatic weekly updates, but they changed often enough that taking your code out for a walk now and then made sense. Now it makes even more sense. See Jon Claerbout’s story along these lines.

The following articles on RR are included.

Guest Editors’ Introduction: Reproducible Research

Sergey Fomel, University of Texas at Austin
Jon F. Claerbout, Stanford University

Reproducible Research in Computational Harmonic Analysis
David L. Donoho, Stanford University
Arian Maleki, Stanford University
Inam Ur Rahman, Apple Computer
Morteza Shahram, Stanford University
Victoria Stodden, Harvard University

Python Tools for Reproducible Research on Hyperbolic Problems
Randall J. LeVeque, University of Washington

Distributed Reproducible Research Using Cached Computations
Roger D. Peng, Johns Hopkins Bloomberg School of Public Health
Sandrah P. Eckel, Johns Hopkins Bloomberg School of Public Health

The Legal Framework for Reproducible Scientific Research: Licensing and Copyright
Victoria Stodden, Harvard University

The current system of scientific journals serves well certain constituencies: the advertisers, the journals themselves, and the authors. It is the underlying philosophy of BioMed Critical Commentary to serve the readers in preference to any other constituency.

In particular, this site could serve as a public forum for criticism that journals are not eager to publish. It could be a good place to discuss specific examples of irreproducible analyses.

The talk presented a study that supposedly discovered genes that can distinguish those who will respond to a certain therapy from those who will not. On closer analysis, the paper actually demonstrated that is it possible to distinguish microarray experiments conducted on one day from experiments conducted another day. That is, batch effects from the lab were much larger than differences between patients who did and did not respond to therapy. I hear that this is typical unless gene expression levels vary dramatically between subgroups.

The talk also discussed problems with reproducing the statistical analysis. As is so often the case, data were mislabeled. In fact, 3/4 of the samples were mislabeled. Simply keeping up with indexes is the biggest barrier to reproducibility. It is shocking how often studies simply did not analyze the data they say they analyzed. This seems like a simple matter to get right; perhaps people give little attention to it precisely because it seems so simple.

So, three reasons to be skeptical of microarray experiment conclusions:

1. High probability of false discovery
2. Statistical reproducibility problems
3. Physical reproducibility problems

Much heat has been generated over the question of whether scientists should be forced to make their data and methodology public.  Journals such as Science and Nature have adopted data dissemination policies; the National Institutes of Health requires data sharing plans for some of its grants; and the Office of Management and Budget Circular A-110 requires that data generated under federally sponsored research be made available upon request if those data were used in developing a government agency action.  While the debate over such dissemination policies is highly relevant, I think it can obscure and cause people to overlook an important question related to reproducible research.

One way I sometimes think of this question is as follows: Suppose a collaborator comes to you and says “I desperately want to make my research reproducible.  What should I do?”  I don’t mean to frame this as purely a hypothetical question—I have actually had people ask me this before.

The problem right now is that I don’t think proponents of reproducible research (myself included) have a good answer to this question.  A typical response might be “make the code and data available”.  Yes, but how?  If we cannot come up with a concrete and coherent answer to this question for people who are willing to make their work reproducible, we cannot realistically expect to change the minds of people who are currently unwilling to make their research reproducible.

I think there are two important roadblocks that make it difficult to publish reproducible research.  The first is the lack of a broad toolset that a wide range of researchers can use to assist them in publishing their data and methodology.  There are a number of efforts out there to develop tools, but many of these tools either have important limitations or are only accessible to more sophisticated users (the Sweave/LaTeX combination comes to mind, although it is a great contribution).  A related problem involves getting people to use tools that are already out there.  For example, I believe the use of version control software is a critical aspect of reproducible research and there are many high-quality software packages available for all operating systems.  I personally use git but many others would also fit the bill.  I must say I’ve had limited success convincing people they need to use version control systems.  I think the basic problem is that it involves learning Yet Another Software Package.

The second roadblock for reproducible research is distribution.  Suppose I carefully keep track of all the code I use to analyze my data and am happy to give the code and data to others.  How do I do that?  Many knowledgeable people will setup a web site for themselves and post code and data on their own web pages.  But demanding that everyone create a web site for distributing reproducible research is in my opinion a steep demand.  Many researchers do not have this capability and even if they did, it is not clear to me that web pages are the ideal medium for disseminating reproducible research.  How much data analysis is done in your web browser?

The distribution problem can be addressed by creating some basic infrastructure.  Analogous infrastructure already exists in other domains.  Users of the R statistical system have the Comprehensive R Archive Network (CRAN) which is used to disseminate R packages (add-on functionality) to anyone around the world.  In practice there is no need to interact with the web site with a browser because R itself can fetch the packages from the Archive and install them without the user ever having to change applications.  Similar facilities exist for Perl (CPAN) and TeX (CTAN).  Of course, we cannot expect such resources to appear out of thin air.  Developing a useful archive requires hardware and administrative time.

I have been trying to develop a system for R users that can be used to distribute reproducible research via a central repository.  The software is an R package called ‘cacher‘ and the associated repository is what I call the Reproducible Research Archive.  The basic idea of the ‘cacher’ package is to take code that represents a data analysis and cache the code and associated data in a series of key-value databases.  This “data analysis cache” can then be packaged and uploaded to the Archive.  Each cache package is given a unique ID (via SHA-1) so that it can be referenced by others in a global fashion.  On the other side of things, the ‘cacher’ package can download an available cache package and a user can run the code in the package to reproduce the results.

Not all of the abovementioned functionality is complete but many aspects of the ‘cacher’ package are available.  There is also a paper in the Journal of Statistical Software that describes the package in greater detail.  The advantage of the ‘cacher’ system is that R users have relatively little to learn—just a few functions.  Of course, the disadvantage is that it is only available to R users, who are a minority of people conducting data analysis in the world.

There are of course other challenges that I haven’t mentioned that will need to be solved before reproducible research goes mainstream. I think the development of the necessary infrastructure (software and distribution media) is just one important challenge that is critical to its adoption because less technical users need to be able to easily “plug-in” to an existing framework without having to build a piece of it themselves.  By learning from experiences in other domains I think we can successfully build this infrastructure and bring reproducible research to a much wider audience.

Roger D. Peng
Department of Biostatistics
Johns Hopkins Bloomberg School of Public Health

Sergey Fomel and Sünje Dallmeier-Tiessen gave presentations in geophysics. Patrick Vandewalle and Jelena Kovacevic gave presentations in signal processing. Mark Liberman, Kai von Fintel, and Steven Krauwer gave presentations related to language and technology.

Video of the presentations is available here.