31
$\begingroup$

Context: In response to an earlier question about reproducible research Jake wrote

One problem we discovered when creating our JASA archive was that versions and defaults of CRAN packages changed. So, in that archive, we also include the versions of the packages that we used. The vignette based system will probably break as folks change their packages (not sure how to include extra packages within the package that is the Compendium).

Finally, I wonder about what to do when R itself changes. Are there ways to produce, say, a virtual machine that reproduces the entire computational environment used for a paper such that the virtual machine is not enormous?

Question:

  • What are good strategies for ensuring that reproducible data analysis is reproducible in the future (say, five, ten, or twenty years after publication)?
  • Specifically, what are good strategies for maximising ongoing reproducibility when using Sweave and R?

This seems to be related to the issue of ensuring that a reproducible data analysis project will run on someone else's machine with slightly different defaults, packages, etc.

Improve this question
$\endgroup$
1
  • $\begingroup$ Have you consider Unit Testing with RUnit to verify theoretical behavior? $\endgroup$
    – user2935
    Commented Jan 26, 2011 at 22:17

6 Answers 6

Reset to default
18
$\begingroup$

At some level, this becomes impossible. Consider the case of the famous Pentium floating point bug: you not only need to conserve your models, your data, your parameters, your packages, all external packages, the host system or language (say, R) as well as the OS ... plus potentially the hardware it all ran on. Now consider that some results may be simulation based and required a particular cluster of machines...

That's just a bit much for being practical.

With that said, I think more pragmatic solutions of versioning your code (and maybe also your data) in revisions control, storing versions of all relevant software and making it possible to reproduce the results by running a single top-level script may be a "good enough" compromise.

Your mileage may vary. This also differs across disciplines or industry. But remember the old saw about the impossibility of foolproof systems: you merely create smarter fools.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ (+1) I can only agree with you. About R specifically, it seems very difficult to ensure that (a) some computations will remain reproducible after updating a package (which happens to me recently), and (b) no conflict with dependencies will emerge one day (it was the case e.g., for lme4). $\endgroup$
    – chl
    Commented Nov 12, 2010 at 10:03
13
$\begingroup$

The first step in reproducibility is making sure the data are in a format that is easy for future researchers to read. Flat files are the clear choice here (Fairbairn in press).

To make the code useful over the long term, perhaps the best thing to do is write clear documentation that explains both what the code does and also how it works, so that if your tool chain disappears, your analysis can be reimplemented in some future system.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Agreed, solid data and metadata first. $\endgroup$
    – mindless.panda
    Commented Nov 12, 2010 at 17:04
12
$\begingroup$

One strategy involves using the cacher package.

  • Peng RD, Eckel SP (2009). "Distributed reproducible research using cached computations," IEEE Computing in Science and Engineering, 11 (1), 28–34. (PDF online)
  • also see more articles on Roger Peng's website

Further discussion and examples can be found in the book:

However, I don't have first hand experience of its effectiveness in ensuring ongoing reproducibility.

Improve this answer
$\endgroup$
0
7
$\begingroup$

If you are interested in the virtual machine route, I think it would be doable via a small linux distribution with the specific version of R and packages installed. Data is included, along with scripts, and package the whole thing in a virtual box file.

This does not get around hardware problems mentioned earlier such as the Intel CPU bug.

Improve this answer
$\endgroup$
4
$\begingroup$

I would recomend two things in addition to the excellent answers already present;

  • At Key points in your code, dump out the current data as a flat file, suitably named and described in comments, thus highlighting if one package has produced differing results where the differences have been introduced. These data files, as well as the original input and the resulting output should be included in your 'reproducible research set'

  • Include some testing of the packages concerned within your code, for instance using something like TestThat. The hard part is making small, reproducible tests that are likely to highlight any changes in what a package does that relate to your analysis. This would at least highlight to another person that there is some difference in the environments.

Improve this answer
$\endgroup$
2
$\begingroup$

Good suggestions, I've got plenty of things to look into now.

Remember, one extremely important consideration is making sure that the work is "correct" in the first place. This is the role that tools like Sweave play, by increasing the chances that what you did, and what you said you did, are the same thing.

Improve this answer
$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.