Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

We often hear of project management and design patterns in computer science, but less frequently in statistical analysis. However, it seems that a decisive step toward designing an effective and durable statistical project is to keep things organized.

I often advocate the use of R and a consistent organization of files in separate folders (raw data file, transformed data file, R scripts, figures, notes, etc.). The main reason for this approach is that it may be easier to run your analysis later (when you forgot how you happened to produce a given plot, for instance).

What are the best practices for statistical project management, or the recommendations you would like to give from your own experience? Of course, this applies to any statistical software. (one answer per post, please)

share|improve this question

6 Answers 6

up vote 63 down vote accepted

I am compiling a quick series of guidelines I found on SO (as suggested by @Shane), Biostar (hereafter, BS), and this SE. I tried my best to acknowledge ownership for each item, and to select first or highly upvoted answer. I also added things of my own, and flagged items that are specific to the [R] environment.

Data management

  • Create a project structure for keeping all things at the right place (data, code, figures, etc., giovanni /BS)
  • Never modify raw data files (ideally, they should be read-only), copy/rename to new ones when making transformations, cleaning, etc.
  • Check data consistency (whuber /SE)
  • Manage script dependencies and data flow with a build automation tool, like GNU make (Karl Broman/Zachary Jones)

Coding

  • organize source code in logical units or building blocks (Josh Reich/hadley/ars /SO; giovanni/Khader Shameer /BS)
  • separate source code from editing stuff, especially for large project -- partly overlapping with previous item and reporting
  • Document everything, with e.g. [R]oxygen (Shane /SO) or consistent self-annotation in the source file -- a good discussion on Medstats, Documenting analyses and data edits Options
  • [R] Custom functions can be put in a dedicated file (that can be sourced when necessary), in a new environment (so as to avoid populating the top-level namespace, Brendan OConnor /SO), or a package (Dirk Eddelbuettel/Shane /SO)

Analysis

  • Don't forget to set/record the seed you used when calling RNG or stochastic algorithms (e.g. k-means)
  • For Monte Carlo studies, it may be interesting to store specs/parameters in a separate file (sumatra may be a good candidate, giovanni /BS)
  • Don't limit yourself to one plot per variable, use multivariate (Trellis) displays and interactive visualization tools (e.g. GGobi)

Versioning

  • Use some kind of revision control for easy tracking/export, e.g. Git (Sharpie/VonC/JD Long /SO) -- this follows from nice questions asked by @Jeromy and @Tal
  • Backup everything, on a regular basis (Sharpie/JD Long /SO)
  • Keep a log of your ideas, or rely on an issue tracker, like ditz (giovanni /BS) -- partly redundant with the previous item since it is available in Git

Editing/Reporting

As a side note, Hadley Wickham offers a comprehensive overview of R project management, including reproducible exemplification and an unified philosophy of data.

Finally, in his R-oriented Workflow of statistical data analysis Oliver Kirchkamp offers a very detailed overview of why adopting and obeying a specific workflow will help statisticians collaborate with each other, while ensuring data integrity and reproducibility of results. It further includes some discussion of using a weaving and version control system. Stata users might find J. Scott Long's The Workflow of Data Analysis Using Stata useful too.

share|improve this answer
    
Great job chl! Would it be o.k. by you if I where to publish this on my blog? (I mean, this text is cc, so I could, but I wanted you permission any way :) ) Cheers, Tal –  Tal Galili Sep 30 '10 at 14:49
    
@Tal No problem. It's far from being an exhaustive list, but maybe you can aggregate other useful links at a later time. Also, feel free to adapt or reorganize in a better way. –  chl Sep 30 '10 at 15:07
    
+1 This is a nice list. You might consider "accepting this" so that it's always on top; given that it's CW, anyone can keep it updated. –  Shane Sep 30 '10 at 15:34
    
@Shane Well, I am indebted to you for providing a first answer with so useful links. Feel free to add/modify the way you want. –  chl Sep 30 '10 at 15:45
    
I republished it here. Great list! r-statistics.com/2010/09/… –  Tal Galili Sep 30 '10 at 16:03

This doesn't specifically provide an answer, but you may want to look at these related stackoverflow questions:

You may also be interested in John Myles White's recent project to create a statistical project template.

share|improve this answer
    
Thanks for the links! The question is open to any statistical software -- I use Python and Stata from time to time, so I wonder if confirmed users may bring interesting recommendations there. –  chl Sep 20 '10 at 20:51
    
Absolutely; although I would add that the recommendations in the above links could really apply to any statistical project (regardless of the language). –  Shane Sep 20 '10 at 20:58
    
Definitely, yes! I updated my question at the same time. –  chl Sep 20 '10 at 21:03

This overlaps with Shane's answer, but in my view there are two main piers:

  • Reproducibility; not only because you won't end with results that are made "somehow" but also be able to rerun the analysis faster (on other data or with slightly changed parameters) and have more time to think about the results. For a huge data, you can first test your ideas on some small "playset" and then easily extend on the whole data.
  • Good documentation; commented scripts under version control, some research journal, even ticket system for more complex projects. Improves reproducibility, makes error tracking easier and writing final reports trivial.
share|improve this answer
    
+1 I like the second point (I use roxygen + git). The first point makes me think also of the possibility to give your code to another statistician that will be able to reproduce your results at a later stage of the project, without any help. –  chl Sep 26 '10 at 9:52

van Belle is the source for the rules of successful statistical projects.

share|improve this answer

Just my 2 cents. I've found Notepad++ useful for this. I can maintain separate scripts (program control, data formatting, etc.) and a .pad file for each project. The .pad file call's all the scripts associated with that project.

share|improve this answer
3  
You mean, notepad++ with the use of npptor :) –  Tal Galili Oct 2 '10 at 8:33

While the other answers are great, I would add another sentiment: Avoid using SPSS. I used SPSS for my master's thesis and now on my regular job in market research.

While working with SPSS, it was incredibly hard to develop organized statistical code, due to the fact that SPSS is bad at handling multiple files (sure, you can handle multiple files , but it's not as painless as R), because you cannot store datasets to a variable - you have to use "dataset activate x"- code, which can be a total pain. Also, the syntax is clunky and encourages shorthands, which make code even more unreadable.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.