Efficient data flow for R?

Question

There are already some good answers on project management (eg. How to efficiently manage a statistical analysis project?). These are great, and make life a lot easier. In particular, the workflow that runs along the lines of

1. Load raw data.
2. Manipulate raw data into useful forms. (Unload raw data to save space).
3. Perform analysis and store results.
4. Make and store figures.

One thing that's still hard is to get from raw data to final results and figures, without having to run the whole project from scratch every time something changes. What I would like to be able to do is something like this:

• Attempt to create figure. Is manipulated data is available in memory?
  • Yes: create figure.
  • No: Does manipulated data exist on disk?
    • Yes: Load manipulated data.
    • No: Load raw data, manipulate, and save, then unload raw data objects.

Also, in each step it would be nice to have some trigger to force a full re-load, if the raw data has been updated.

Is there an existing framework in R for doing something like this? Or are there any recommended ways of doing this? As it stands, I often have to run everything from scratch, which can take ages (large data files, complex manipulations), and is a waste of resources

Answer 1

R can store all data from the memory in the current working directory. Then, when you start R from a particular directory, it will load all the data that you had when you quit R last time.

mkdir ProjectX
cd ProjectX
R

Then, work in R. When R quits, it asks

Save workspace image? [y/n/c]:

I choose y. History is saved to a file called .Rhistory, and all the data in .RData. Next time I start R in the same directory, it will load both the history and the data.

Usually, I create a directory for a project and work with R in that directory. I store all commands in the per-directory command line history, but I also create a file that can be directly run to re-create all calculations and figures. I have found no more efficient way than stitching this file manually from the R history.

If I modify some of the raw data, I can simply do

source( "pipeline.R" )

to repeat all the commands that were needed to run the calculations and produce the figures.

Otherwise, if only minor modifications were done, I either search through the history or copy-and-paste from the pipeline.R file.