7
$\begingroup$

Context:

I've recently adopted version control as part of my data analysis work (finally I may hear you saying: see my earlier question on SO). This prompted me to think more about repositories and the directory structure I use for my projects.

My typical research work involves one or more studies (i.e., data that I have collected) which gets written up as one or more publications (journal articles, book chapters, presentations, reports, etc.). Analyses and reports are typically produced using a combination of R, LaTeX, Sweave, textual data files and so on. I really like the idea of being able to upload a single self-contained repository that can be used to analyse data and reproduce a publication.

In particular, I've been thinking about publications, studies, data, and common code, and how these entities map on to repositories. For example, is it better to have a separate repository for each publication or is it better to have each publication as an individual folder within the larger repository. I'm evolving a few thoughts on this, but I was keen to hear other options.

Question:

  • What strategies do people use to map studies, publications, and analyses onto repositories?
  • When should related entities (e.g., publications, studies, etc.) be split into multiple repositories?
Improve this question
$\endgroup$

2 Answers 2

Reset to default
5
$\begingroup$

Regarding your first question:

What strategies do people use to map studies, publications, and analyses onto repositories?

One year ago or so, I decided to have one repository for each publication, presentation or semester/class. My typical directory looks like this:

.git
.gitignore
README.org
ana
dat
doc
org

The underlying idea is (hopefully) obvious: Each publication, presentation, class is an "autarchic entity" which I could easily share with others.

By the way, you are not the first to ask this question: One repository/multiple projects without getting mixed up?

However, I also started using git for managing projects which might result in some publications (the directory structure follows roughly John Myles White's ProjectTemplate, without using it, though).

.git
.gitignore
README.org
ana
data
docs
graphs
lib
org
reports
tests

Regarding your second question:

When should related entities (e.g., publications, studies, etc.) be split into multiple repositories?

I cannot think of any reason to split a publication, conference etc. related repository into multiple repositories. But I would be interested in other opinions...

Improve this answer
$\endgroup$
7
  • $\begingroup$ @Bernd. Thanks for that. With regards to the second question, I was thinking of several scenarios. For example, you conduct two studies and they get written up in a journal article and conference proceedings. Then you do a third study and present a second conference paper using all three studies. Perhaps you can see where I'm going with this. There are connections between the set of studies and the set of publications and presentations. Should they all be in one repository, or would you split them up in some way (e.g., one repository per publication)? $\endgroup$
    – Jeromy Anglim
    Commented Nov 14, 2010 at 8:18
  • $\begingroup$ @Bernd; thanks also for the link to the SO question. It suggests having one project per repository. However, for me it still doesn't quite answer the question: "what is a project?" In particular, it seems to me that there are a few many-to-many relationships going on between studies and publications that can sometimes complicate things. $\endgroup$
    – Jeromy Anglim
    Commented Nov 14, 2010 at 8:21
  • $\begingroup$ (Sorry, just realized that comments are limited to 600 characters) @Jeromy (part 1/3): You wrote: "There are connections between the set of studies and the set of publications and presentations. Should they all be in one repository, or would you split them up in some way (e.g., one repository per publication)?" When you write "split them up in some way", do you mean sth like forking? And "studies" are datasets? As I wrote before, each publication, each presentation etc. has its own repository. $\endgroup$
    – Bernd Weiss
    Commented Nov 14, 2010 at 11:04
  • $\begingroup$ @Jeromy (part 2/3): However, there is one real downside: using the same dataset in different publications means to maintain different versions of "initialization code" (define missing values, generate new variables etc.). To overcome this problem, I decided to maintain /one/ study/dataset-related repository which contains the original init-file. For each publication, presentation etc. I am using a copy of the original data-file as well as of the init-file (in R via file.copy()). $\endgroup$
    – Bernd Weiss
    Commented Nov 14, 2010 at 11:05
  • $\begingroup$ @Jeromy (part 3/3): Of course, whenever I create a new variable I need to modify my original init-file and do a file.copy() (which is the most annoying part of my approach). Regarding your question "what is a project?", I would say that project = (primary) study/dataset. $\endgroup$
    – Bernd Weiss
    Commented Nov 14, 2010 at 11:06
2
$\begingroup$

I keep a separate repository for each project, with a project being centered around a particular data set or question being addressed. The repo contains the data, code, and Sweave documents/plots that explain and express the results.

I maintain a separate repo for each discrete publication or presentation because

  1. A single project may result in multiple publications or presentations. Once a publication is out or you've given the presentation, you are essentially "done" with the contents of that repo so they don't need to be dragged around with the project.
  2. A output (publication/presentation/chapter) may contain data from more than one project.
  3. Not all of the results from a project will end up in a particular piece of output.

Code that is reusable across projects gets its own repo, as well. If I come up with a new & discrete question using data that is already in one repo, I'll copy that data to a new repo.

If you want to be really strict about it, many version control systems offer the idea of "subprojects", but I've found that to be overkill.

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.