How are we defining 'reproducible research'?

Question

This has come up in a few questions now, and I've been wondering about something. Has the field as a whole moved toward "reproducibility" focusing on the availability of the original data, and the code in question?

I was always taught that the core of reproducibility was not necessarily, as I've referred to it, the ability to click Run and get the same results. The data-and-code approach seems to assume that the data are correct - that there isn't a flaw in the collection of the data itself (often demonstrably false in the case of scientific fraud). It also focuses on a single sample of the target population, rather than the replicability of the finding over multiple independent samples.

Why is the emphasis then on being able to re-run analysis, rather than duplicate the study from the ground up?

The article mentioned in the comments below is available here.

Answer 1

"Reproducible research" as reproducible analysis

Reproducible research is a term used in some research domains to refer specifically to conducting analyses such that

• code transforms raw data and meta-data into processed data,
• code runs analyses on the data, and
• code incorporates analyses into a report.

When such data and code are shared, this allows other researchers to:

• perform analyses not reported by the original researchers
• check the correctness of the analyses performed by the original researchers

This usage can be seen in discussions of technologies like Sweave. E.g., Friedrich Leisch writes in the context of Sweave that "the report can be automatically updated if data or analysis change, which allows for truly reproducible research." It can also be seen in the CRAN Task View on Reproducible Research which states that "the goal of reproducible research is to tie specific instructions to data analysis and experimental data so that scholarship can be recreated, better understood and verified."

Broader usage of the term "reproducibility"

Reproducibility is a fundamental aim of science. It's not new. Research reports include method and results sections that should outline how the data was generated, processed, and analysed. A general rule is that the details provided should be sufficient to enable an appropriately competent researcher to take the information provided and replicate the study.

Reproducibility is also closely related to the concepts of replicability and generalisation.

Thus, the term "reproducible research", taken literally, as applied to technologies like Sweave, is a misnomer, given that it suggests a relevance broader than it covers. Also, when presenting technologies like Sweave to researchers who have not used such technologies, such researchers are often surprised when I call the process "reproducible research".

A better term than "reproducible research"

Given that "reproducible research" as used within Sweave-like contexts only pertains to one aspect of reproducible research, perhaps an alternative term should be adopted. Possible alternatives include:

• Reproducible analysis:
  • John D Cook has used this term
  • Jennifer Blackford uses the term "reliable and reproducible analyses"
• Reproducible data analysis
  • Christophe Pouzat uses this term
• Reproducible statistical analysis
  • A Biostats site at Vanderbilt uses the term "reproducible statistical analysis and reporting activities"
• Reproducible reporting

All of the above terms are a more accurate reflection of what Sweave-like analyses entail. Reproducible analysis is short and sweet. Adding "data" or "statistical" further clarifies things, but also makes the term both longer and narrower. Furthermore, "statistical" has a narrow and a broad meaning, and certainly within the narrow meaning, much of data processing is not statistical. Thus, the breadth implied by the term "reproducible analysis" has its advantages.

It's not just about reproducibility

The other additional issue with the term "reproducible research" is the aim of Sweave-like technologies is not just "reproducibility". There are several interrelated aims:

• Reproducibility
  • Can the analyses easily be re-run to transform raw data into final report with the same results?
• Correctness
  • Is the data analysis consistent with the intentions of the researcher?
  • Are the intentions of the researcher correct?
• Openness
  • Transparency, accountability
    • Can others check and verify the accuracy of analyses performed?
  • Extensibility, modfifiability
    • Can others modify, extend, reuse, and mash, the data, analyses, or both to create new research works?

There is an argument that reproducible analysis should promote correct analyses, because there is a written record of analyses that can be checked. Furthermore if data and code is shared, it creates accountability which motivates researchers to check their analyses, and enables other researchers to note corrections.

Reproducible analysis also fits in closely with concepts around open research. Of course, a researcher can use Sweave-like technologies just for themselves. Open research principles encourage sharing the data and analysis code to enable greater reuse and accountability.

This is not really a critique of the use of the word "reproducible". Rather, it just highlights that using Sweave-like technologies is necessary but not sufficient to achieving open scientific research aims.

Answer 2

Having access to the data and code for the analysis in an easy-to-execute form is a sine qua non of reproducible research. Once you verify that the analysis works, you can substitute your own code/data where you are skeptical of the original author's. I'd say that the majority of statistics-containing papers I read have at least one part of the methodology that is left vague. My attempts to reproduce these analyses are often unsuccessful (and always time-consuming), but it is very difficult to say whether this is because of fraud, human error, or (much more likely) my resolving these ambiguities differently than the author. So, having data+code for a paper does not guarantee that its conclusions are true, but it makes it much easier to critique or extend them.

Also, "reproducible research" a matter of degree. So the reproducible research movement can be seen as encouraging research that is "more reproducible" than the norm, rather than demanding that research meet some minimum threshold. I'd guess that "release the data and code" is in vogue now because it is a relatively easy and non-threatening step.

Answer 3

Being able to re-run everything is a starting point for the reproducible research. It permits to show that you are actually using the same procedure. After that -and only after that- you can pursue the research of your peer. In other words, the strict reproducibility is not to be perceived as a time at which the research is moving forward, but as a landmark, a consensus, something on which people people agree. Isn't thi fundamental to get further ?

Also, according to the discussion of Donoho (read section 2 "the scandal") the aim of reproducible research is also to test robustness of the given code. First by playing with the code, making sligth modification that was not done in the paper (because we don't want papers with 30 Figures ...). I think the concept of reproducible research in the litterature contains the idea of having strong robust landmark. It almost contain the idea of going further.