92
$\begingroup$

We often hear of project management and design patterns in computer science, but less frequently in statistical analysis. However, it seems that a decisive step toward designing an effective and durable statistical project is to keep things organized.

I often advocate the use of R and a consistent organization of files in separate folders (raw data file, transformed data file, R scripts, figures, notes, etc.). The main reason for this approach is that it may be easier to run your analysis later (when you forgot how you happened to produce a given plot, for instance).

What are the best practices for statistical project management, or the recommendations you would like to give from your own experience? Of course, this applies to any statistical software. (one answer per post, please)

$\endgroup$
4
  • $\begingroup$ I'm voting to close this question as off-topic because it's about project management $\endgroup$
    – Aksakal
    Aug 9, 2016 at 21:23
  • 2
    $\begingroup$ @Aksakal: I think you are a bit harsh. :) It is relevant to "people interested in statistics". Also the 70+ votes strongly suggest standard users found this question of interest and useful. $\endgroup$
    – usεr11852
    Aug 9, 2016 at 22:25
  • 2
    $\begingroup$ I think this should be considered on topic here. $\endgroup$ Aug 9, 2016 at 23:58
  • $\begingroup$ @gung Would you perhaps like to add an answer to that Meta thread so that we could discuss it? $\endgroup$
    – amoeba
    Aug 10, 2016 at 15:24

7 Answers 7

81
$\begingroup$

I am compiling a quick series of guidelines I found on SO (as suggested by @Shane), Biostar (hereafter, BS), and this SE. I tried my best to acknowledge ownership for each item, and to select first or highly upvoted answer. I also added things of my own, and flagged items that are specific to the [R] environment.

Data management

  • Create a project structure for keeping all things at the right place (data, code, figures, etc., giovanni /BS)
  • Never modify raw data files (ideally, they should be read-only), copy/rename to new ones when making transformations, cleaning, etc.
  • Check data consistency (whuber /SE)
  • Manage script dependencies and data flow with a build automation tool, like GNU make (Karl Broman/Zachary Jones)

Coding

  • organize source code in logical units or building blocks (Josh Reich/hadley/ars /SO; giovanni/Khader Shameer /BS)
  • separate source code from editing stuff, especially for large project -- partly overlapping with previous item and reporting
  • Document everything, with e.g. [R]oxygen (Shane /SO) or consistent self-annotation in the source file -- a good discussion on Medstats, Documenting analyses and data edits Options
  • [R] Custom functions can be put in a dedicated file (that can be sourced when necessary), in a new environment (so as to avoid populating the top-level namespace, Brendan OConnor /SO), or a package (Dirk Eddelbuettel/Shane /SO)

Analysis

  • Don't forget to set/record the seed you used when calling RNG or stochastic algorithms (e.g. k-means)
  • For Monte Carlo studies, it may be interesting to store specs/parameters in a separate file (sumatra may be a good candidate, giovanni /BS)
  • Don't limit yourself to one plot per variable, use multivariate (Trellis) displays and interactive visualization tools (e.g. GGobi)

Versioning

  • Use some kind of revision control for easy tracking/export, e.g. Git (Sharpie/VonC/JD Long /SO) -- this follows from nice questions asked by @Jeromy and @Tal
  • Backup everything, on a regular basis (Sharpie/JD Long /SO)
  • Keep a log of your ideas, or rely on an issue tracker, like ditz (giovanni /BS) -- partly redundant with the previous item since it is available in Git

Editing/Reporting

As a side note, Hadley Wickham offers a comprehensive overview of R project management, including reproducible exemplification and an unified philosophy of data.

Finally, in his R-oriented Workflow of statistical data analysis Oliver Kirchkamp offers a very detailed overview of why adopting and obeying a specific workflow will help statisticians collaborate with each other, while ensuring data integrity and reproducibility of results. It further includes some discussion of using a weaving and version control system. Stata users might find J. Scott Long's The Workflow of Data Analysis Using Stata useful too.

$\endgroup$
12
  • $\begingroup$ Great job chl! Would it be o.k. by you if I where to publish this on my blog? (I mean, this text is cc, so I could, but I wanted you permission any way :) ) Cheers, Tal $\endgroup$
    – Tal Galili
    Sep 30, 2010 at 14:49
  • $\begingroup$ @Tal No problem. It's far from being an exhaustive list, but maybe you can aggregate other useful links at a later time. Also, feel free to adapt or reorganize in a better way. $\endgroup$
    – chl
    Sep 30, 2010 at 15:07
  • $\begingroup$ +1 This is a nice list. You might consider "accepting this" so that it's always on top; given that it's CW, anyone can keep it updated. $\endgroup$
    – Shane
    Sep 30, 2010 at 15:34
  • $\begingroup$ @Shane Well, I am indebted to you for providing a first answer with so useful links. Feel free to add/modify the way you want. $\endgroup$
    – chl
    Sep 30, 2010 at 15:45
  • $\begingroup$ I republished it here. Great list! r-statistics.com/2010/09/… $\endgroup$
    – Tal Galili
    Sep 30, 2010 at 16:03
21
$\begingroup$

This doesn't specifically provide an answer, but you may want to look at these related stackoverflow questions:

You may also be interested in John Myles White's recent project to create a statistical project template.

$\endgroup$
3
  • $\begingroup$ Thanks for the links! The question is open to any statistical software -- I use Python and Stata from time to time, so I wonder if confirmed users may bring interesting recommendations there. $\endgroup$
    – chl
    Sep 20, 2010 at 20:51
  • $\begingroup$ Absolutely; although I would add that the recommendations in the above links could really apply to any statistical project (regardless of the language). $\endgroup$
    – Shane
    Sep 20, 2010 at 20:58
  • $\begingroup$ Definitely, yes! I updated my question at the same time. $\endgroup$
    – chl
    Sep 20, 2010 at 21:03
8
$\begingroup$

This overlaps with Shane's answer, but in my view there are two main piers:

  • Reproducibility; not only because you won't end with results that are made "somehow" but also be able to rerun the analysis faster (on other data or with slightly changed parameters) and have more time to think about the results. For a huge data, you can first test your ideas on some small "playset" and then easily extend on the whole data.
  • Good documentation; commented scripts under version control, some research journal, even ticket system for more complex projects. Improves reproducibility, makes error tracking easier and writing final reports trivial.
$\endgroup$
2
  • $\begingroup$ +1 I like the second point (I use roxygen + git). The first point makes me think also of the possibility to give your code to another statistician that will be able to reproduce your results at a later stage of the project, without any help. $\endgroup$
    – chl
    Sep 26, 2010 at 9:52
  • $\begingroup$ Reproducibility? Data has random error anyway, so who cares. Documentation? Two possible answers: 1) We're too busy, we don't have time for documentation or 2) We only had budget to either do the analysis or document it, so we chose to do the analysis. You think I'm joking? I've seen/heard these attitudes on many occasions - on projects on which lives were riding on the line. $\endgroup$ Aug 10, 2016 at 0:37
4
$\begingroup$

van Belle is the source for the rules of successful statistical projects.

$\endgroup$
1
$\begingroup$

Just my 2 cents. I've found Notepad++ useful for this. I can maintain separate scripts (program control, data formatting, etc.) and a .pad file for each project. The .pad file call's all the scripts associated with that project.

$\endgroup$
1
  • 3
    $\begingroup$ You mean, notepad++ with the use of npptor :) $\endgroup$
    – Tal Galili
    Oct 2, 2010 at 8:33
1
$\begingroup$

While the other answers are great, I would add another sentiment: Avoid using SPSS. I used SPSS for my master's thesis and now on my regular job in market research.

While working with SPSS, it was incredibly hard to develop organized statistical code, due to the fact that SPSS is bad at handling multiple files (sure, you can handle multiple files , but it's not as painless as R), because you cannot store datasets to a variable - you have to use "dataset activate x"- code, which can be a total pain. Also, the syntax is clunky and encourages shorthands, which make code even more unreadable.

$\endgroup$
0
$\begingroup$

Jupyter Notebooks, which work with R/Python/Matlab/etc, remove the hassle of remembering which script generates a certain figure. This post describes a tidy way of keeping the code and the figure right beside each other. Keeping all figures for a paper or thesis chapter in a single notebook makes the asccoiated code very easy to find.

Even better, in fact, because you can scroll through, say, a dozen figures to find the one you want. The code is kept hidden until it is needed.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.