I am looking for input on how others organize their R code and output.

My current practice is to write code in blocks in a text file as such:

#=================================================
# 19 May 2011
date()
# Correlation analysis of variables in sed summary
load("/media/working/working_files/R_working/sed_OM_survey.RData")
# correlation between estimated surface and mean perc.OM in epi samples
cor.test(survey$mean.perc.OM[survey$Depth == "epi"], 
    survey$est.surf.OM[survey$Depth   == "epi"]))
#==================================================

I then paste the output into another text file, usually with some annotation.

The problems with this method are:

  1. The code and the output are not explicitly linked other than by date.
  2. The code and output are organized chronologically and thus can be hard to search.

I have considered making one Sweave document with everything since I could then make a table of contents but this seems like it may be more hassle than the benefits it would provide.

Let me know of any effective routines you have for organizing your R code and output that would allow for efficient searching and editing the analysis.

closed as off-topic by Carlos Cinelli, Peter Flom Nov 11 at 11:25

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – Carlos Cinelli, Peter Flom
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 2
    Just to avoid copy/paste, sink() or capture.output() might be your friends. Reporting utilities, like Hmisc, Sweave, or brew are worth to consider (your point 1). Versioning systems (rcs, svn, or git) might help with point 2. – chl May 19 '11 at 21:53
  • @chl - thanks for the suggestions. I wasn't aware of sink() and capture.output(). Thats great. – KennyPeanuts May 20 '11 at 13:15
  • 1
    today there is also knitr! – kjetil b halvorsen Jul 4 '17 at 22:13
up vote 21 down vote accepted

You are not the first person to ask this question.

  • +1 and the first link you provide references a thread therein :-) – chl May 19 '11 at 21:45
  • @chl Thanks! I was wondering if this question is a duplicate and should be closed... – Bernd Weiss May 19 '11 at 21:54
  • It is, IMO. But as there are no votes to close, I'm reluctant to close it. Also, the older one was more general, but a very similar question has been closed in the past. Let's wait and see how it goes. – chl May 19 '11 at 21:58
  • thanks for the list! This is very useful. I figured I wasn't the first to have this question but I didn't seem to find much with my (obviously inept) initial search. – KennyPeanuts May 20 '11 at 13:01

I for one organize everything into 4 files for every project or analysis. (1) 'code' Where I store text files of R functions. (2) 'sql' Where I keep the queries used to gather my data. (3) 'dat' Where I keep copies (usually csv) of my raw and processed data. (4) 'rpt' Where I store the reports I've distributed.

ALL of my files are named using very verbose names such as 'analysis_of_network_abc_for_research_on_modified_buffer_19May2011'

I also write detailed documentation up front where I organize the hypothesis, any assumptions, inclusion and exclusion criteria, and steps I intend to take to reach my deliverable. All of this is invaluable for repeatable research and makes my annual goal setting process easier.

Now that I've made the switch to Sweave I never want to go back. Especially if you have plots as output, it's so much easier to keep track of the code used to create each plot. It also makes it much easier to correct one minor thing at the beginning and have it ripple through the output without having to rerun anything manually.

  • 1
    Sweave is wonderful. It takes some getting used to, but if you already know TeX and R, its the obvious choice. It also allows you to never spend time aligning table columns ever again, which is nice. – richiemorrisroe May 20 '11 at 9:22
  • thanks for the input. My 2 concerns with Sweave are 1) I will wind up with a bazillion files in my directory - especially with a lot of figures, and 2) I will have to be really careful with the code to prevent hiccups each time I compile the whole doc (e.g., I think something is loaded and its not). Do you have these problems? – KennyPeanuts May 20 '11 at 13:06
  • 1) You can choose to delete the intermediate files if you want; I use the Sweave.sh script (cran.r-project.org/contrib/extra/scripts/Sweave.sh) which does so automatically; though can easily be turned off. If you do that, make sure you know what it will delete before using it to prevent possible disaster. The short version is that if no files share the basename of your Rnw file you're okay. – Aaron May 20 '11 at 13:13
  • 2) In my opinion, having to be careful in that way is a Good Thing, and will sometimes compile my Sweave document with that exact purpose in mind, that is, to make sure I've properly kept track of everything needed to recreate the analysis. – Aaron May 20 '11 at 13:16
  • 1
    @naught101: For long analyses, I do run it separately and save the results, usually in an .RData file, for input by the Sweave document. However, there are also several great options out there for "caching" the results from a code chunk so it doesn't get rerun. – Aaron Jul 20 '12 at 12:16

For structuring single .R code files, you can also use strcode, a RStudio add-in I created to insert code separators (with optional titles) and based on them - obtain summaries of code files. I explain the usage of it in more detail in this blog post.

Not the answer you're looking for? Browse other questions tagged or ask your own question.