74
$\begingroup$

The Question: Are there any good examples of reproducible research using R that are freely available online?

Ideal Example: Specifically, ideal examples would provide:

  • The raw data (and ideally meta data explaining the data),
  • All R code including data import, processing, analyses, and output generation,
  • Sweave or some other approach for linking the final output to the final document,
  • All in a format that is easily downloadable and compilable on a reader's computer.

Ideally, the example would be a journal article or a thesis where the emphasis is on an actual applied topic as opposed to a statistical teaching example.

Reasons for interest: I'm particularly interested in applied topics in journal articles and theses, because in these situations, several additional issues arise:

  • Issues arise related to data cleaning and processing,
  • Issues arise related to managing metadata,
  • Journals and theses often have style guide expectations regarding the appearance and formatting of tables and figures,
  • Many journals and theses often have a wide range of analyses which raise issues regarding workflow (i.e., how to sequence analyses) and processing time (e.g., issues of caching analyses, etc.).

Seeing complete working examples could provide good instructional material for researchers starting out with reproducible research.

Improve this question
$\endgroup$

15 Answers 15

Reset to default
15
$\begingroup$

Frank Harrell has been beating the drum on reproducible research and reports for many, many years. You could start at this wiki page which lists plenty of other resources, including published research and also covers Charles Geyer's page.

Improve this answer
$\endgroup$
12
$\begingroup$

The journal Biostatistics has an Associate Editor for Reproducibility, and all its articles are marked:

Reproducible Research

Our reproducible research policy is for papers in the journal to be kite-marked D if the data on which they are based are freely available, C if the authors’ code is freely available, and R if both data and code are available, and our Associate Editor for Reproducibility is able to use these to reproduce the results in the paper. Data and code are published electronically on the journal’s website as Supplementary Materials.

http://biostatistics.oxfordjournals.org/

How good an idea is that?

http://biostatistics.oxfordjournals.org/content/12/1/18.abstract comes with an R package in the supplementaries that does the analysis - haven't tried it myself yet. Also, can't find out where the openness rating is specified. Am emailing the associate editor with some questions...

[edit]

Roger Peng the associate editor tells me there probably is no way of finding the reproducible papers without getting the PDF. He pointed me at this one which has a nice big 'R' on it (which does not mean R-rated like movies) for reproducibility:

http://biostatistics.oxfordjournals.org/content/10/3/409.abstract

Of course the journal itself isn't free... #fail

Barry

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ that's great to see a journal prioritising reproducibility. Have you seen any good examples of articles marked R? $\endgroup$
    – Jeromy Anglim
    Commented Jul 28, 2011 at 8:41
  • 1
    $\begingroup$ They don't prioritise it for publication, I think they just want to highlight it. I'll edit my answer with an example. $\endgroup$
    – Spacedman
    Commented Jul 28, 2011 at 10:44
10
$\begingroup$

Irreproducibility of NCI60 Predictors of Chemotherapy

This is a reproducible analysis showing the lack of reproducibility of a paper that has been in the news. A clinical trial based on the false conclusions of the irreproducible paper was suspended, re-instated, suspended again, ... It's a good example of reproducible analysis in the news.

Improve this answer
$\endgroup$
10
$\begingroup$

I have a few such examples on my research papers page. (I am not allowed to post more than one hyperlink as a new member. So I'll just describe the papers on that site.)

(1) "Making Effects Manifest in Randomized Experiments" uses R's vignette system.

(2) "Attributing Effects to a Cluster Randomized Get-Out-The-Vote Campaign" was a more complex paper involving some time consuming simulations. We used a Makefile based system and posted it to the Dataverse

(3) "EDA for HLM" was my earliest attempt. Here I just put the data and associated Sweave files in a tarball.

One problem we discovered when creating our JASA archive was that versions and defaults of CRAN packages changed. So, in that archive, we also include the versions of the packages that we used. The vignette based system will probably break as folks change their packages (not sure how to include extra packages within the package that is the Compendium).

Finally, I wonder about what to do when R itself changes. Are there ways to produce, say, a virtual machine that reproduces the entire computational environment used for a paper such that the virtual machine is not enormous?

Anyway, I hope that these examples help. At least they show some of my own experiments in this area.

(Here are some plain text hyperlinks.)

  [2]: http://jakebowers.org/manifesteffects-compendium-howto.txt
  [3]: http://hdl.handle.net/1902.1/12174
  [4]: http://hdl.handle.net/1902.1/13376
Improve this answer
$\endgroup$
1
9
$\begingroup$

Koenker and Zeileis provide a webpage with a relatively complete example. They share:

  • Rnw (Sweave code)
  • R analysis code
  • Final PDF
  • Discussion of version control issues
Improve this answer
$\endgroup$
8
$\begingroup$

We wrote a paper explaining how to use R/Bioconductor when analysing microarray data. The paper was written in Sweave and all the code used to generate the graphs is included as supplementary material.

Gillespie, C. S., Lei, G., Boys, R. J., Greenall, A. J., Wilkinson, D. J., 2010. Analysing yeast time course microarray data using BioConductor: a case study using yeast2 Affymetrix arrays BMC Research Notes, 3:81.

Improve this answer
$\endgroup$
7
$\begingroup$

Charles Geyer's page on Sweave has an example from a thesis, which meets some of your requirements (the raw data is simply from an R package, but the R/sweave code and final PDF are available):

A paper on the theory in Yun Ju Sung's thesis, Monte Carlo Likelihood Inference for Missing Data Models (preprint) contained computing examples. Every number in the paper and every plot was taken (by cut-and-paste, I must admit) from a "supplementary materials" document done in Sweave.

(The source file is linked under the "Supplementary Materials for a Paper" section.)

I know I've come across at least one R example browsing the ReproducibleResearch.net material page before, but unfortunately didn't bookmark it.

Improve this answer
$\endgroup$
0
5
$\begingroup$

Neil Saunders analysed online interactions associated with a conference. Several properties which make it a useful Sweave example include:

  • Rnw file is provided
  • Graphs are generated using ggplot
  • Good size and easily comprehensible domain

The materials are available here:

Improve this answer
$\endgroup$
5
$\begingroup$

Simon Jackman has a particularly useful example of analysing the results of a survey: "Americans and Australians 10 years after 9/11". It has multiple examples of integrating tables and figures.

He has made the Sweave document and PDF report via this blog post.

While the raw data is not supplied (as far as I can tell), so it's not possible to run the actual Sweave examples, I think a fair bit can be learned from studying the Sweave code.

Improve this answer
$\endgroup$
0
4
$\begingroup$

Also look at Journal Of Statistical Software; they encourage making papers in Sweave.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ No, not formally -- LaTeX submission is encourages but if you look at the instructions page it does not contain the word Sweave. Authors do use it and/or ship the R code with the paper, but to me this echos Shane's point about package vignettes. $\endgroup$
    – Dirk Eddelbuettel
    Commented Aug 21, 2010 at 11:44
  • 1
    $\begingroup$ Ok, still most submitters do use it (also journal style includes Swave.sty); the main problem is that there are no Rnws published, still papers made by Sweave come with Stangle output. $\endgroup$
    – user88
    Commented Aug 21, 2010 at 13:21
4
$\begingroup$

http://genome.cshlp.org/content/early/2011/06/09/gr.117523.110/suppl/DC1

A nice paper, by a lab mate of mine. Our PI was pretty pleased when something resembling fan mail came in for this. Now all publications from the group have the supplemental methods laid out in LaTeX/Sweave. Some of the papers, too (can't decide whether to keep mine in LyX/Sweave or fold and just do the supplementals in Sweave).

Improve this answer
$\endgroup$
4
$\begingroup$

Robert Gentleman wrote a paper called "Reproducible Research: A Bioinformatics Case Study".

It implements a short set of analyses as an R Package and uses Sweave. It also discusses the use of Sweave more generally.

See the "Related Files" section of the article page for an archive file of all files and folders used.

Reference:

  • Gentleman, Robert (2005) "Reproducible Research: A Bioinformatics Case Study," Statistical Applications in Genetics and Molecular Biology: Vol. 4 : Iss. 1, Article 2.
  • DOI: 10.2202/1544-6115.1034
  • Available here.
Improve this answer
$\endgroup$
4
$\begingroup$

I have found good ones in the past and will post once I dig them up, but some quick general suggestions:

  1. You may be able to find some interesting examples by searching google with keywords and ext:rnw (which will search for files with the sweave extension). Here's an example search. This is the third result from my search: http://www.ne.su.se/paper/araietal_source.Rnw. Here's another example from my search: http://www.stat.umn.edu/geyer/gdor/.
  2. Many R packages have interesting vignettes which essentially amount to the same thing. An example: https://r-forge.r-project.org/scm/viewvc.php/paper/maxLik.Rnw$^\dagger$

$^\dagger$ Not working anymore.

Improve this answer
$\endgroup$
3
$\begingroup$

Looking for examples and practices is a good way to learn, but I just wanted to mention that reproducibility has not only technical/script rerun side, but also code style and structuring aspect, minimization of side effects in core functions etc. I personally found that Chambers book Software for Data Analysis allows to understand more deeply techniques that help to avoid reliability and reproducibility issues on R code level.

Improve this answer
$\endgroup$
2
$\begingroup$

if you still need a great example of a fully REPRODUCIBLE analysis plus a PAPER, use this repo.

The @jscamac did a great job by making his analysis rproducible and I personally validated it.

You can lean how to use R specific functions like the package remake to ensure reproduciblity.

Watch out / the calculations take about one hour to complete.

Its all scripted and produces a LaTeX paper in the end with figures.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.