How to increase longer term reproducibility of research (particularly using R and Sweave) Context:
In response to an earlier question about reproducible research Jake wrote

One problem we discovered when
  creating our JASA archive was that
  versions and defaults of CRAN packages
  changed. So, in that archive, we also
  include the versions of the packages
  that we used. The vignette based
  system will probably break as folks
  change their packages (not sure how to
  include extra packages within the
  package that is the Compendium).
Finally, I wonder about what to do
  when R itself changes. Are there ways
  to produce, say, a virtual machine
  that reproduces the entire
  computational environment used for a
  paper such that the virtual machine is
  not enormous?

Question:


*

*What are good strategies for ensuring that reproducible data analysis is reproducible in the future (say, five, ten, or twenty years after publication)?

*Specifically, what are good strategies for maximising ongoing reproducibility when using Sweave and R?


This seems to be related to the issue of ensuring that a reproducible data analysis project will run on someone else's machine with slightly different defaults, packages, etc.
 A: If you are interested in the virtual machine route, I think it would be doable via a small linux distribution with the specific version of R and packages installed.  Data is included, along with scripts, and package the whole thing in a virtual box file.
This does not get around hardware problems mentioned earlier such as the Intel CPU bug.
A: I would recomend two things in addition to the excellent answers already present;


*

*At Key points in your code, dump out the current data as a flat file, suitably named and described in comments, thus highlighting if one package has produced differing results where the differences have been introduced.  These data files, as well as the original input and the resulting output should be included in your 'reproducible research set'

*Include some testing of the packages concerned within your code, for instance using something like TestThat.  The hard part is making small, reproducible tests that are likely to highlight any changes in what a package does that relate to your analysis.  This would at least highlight to another person that there is some difference in the environments.
A: At some level, this becomes impossible.  Consider the case of the famous Pentium floating point bug: you not only need to conserve your models, your data, your parameters, your packages, all external packages, the host system or language (say, R) as well as the OS ... plus potentially the hardware it all ran on.  Now consider that some results may be simulation based and required a particular cluster of machines...
That's just a bit much for being practical.
With that said, I think more pragmatic solutions of versioning your code (and maybe also your data) in revisions control, storing versions of all relevant software and making it possible to reproduce the results by running a single top-level script may be a "good enough" compromise.  
Your mileage may vary. This also differs across disciplines or industry.  But remember the old saw about the impossibility of foolproof systems: you merely create smarter fools.
A: The first step in reproducibility is making sure the data are in a format that is easy for future researchers to read. Flat files are the clear choice here (Fairbairn in press).
To make the code useful over the long term, perhaps the best thing to do is write clear documentation that explains both what the code does and also how it works, so that if your tool chain disappears, your analysis can be reimplemented in some future system. 


*

*Fairbairn (in press) The advent of mandatory data archiving. Evolution DOI: 10.1111/j.1558-5646.2010.01182.x

A: One strategy involves using the cacher package.


*

*Peng RD, Eckel SP (2009). "Distributed reproducible research using cached computations," IEEE Computing in Science and Engineering, 11 (1), 28–34. (PDF online)

*also see more articles on
Roger Peng's website
Further discussion and examples can be found in the book:


*

*Statistical Methods for Environmental Epidemiology with R
However, I don't have first hand experience of its effectiveness in ensuring ongoing reproducibility.
A: Good suggestions, I've got plenty of things to look into now.  
Remember, one extremely important consideration is making sure that the work is "correct" in the first place.  This is the role that tools like Sweave play, by increasing the chances that what you did, and what you said you did, are the same thing.
