# How to keep exploratory analyses of large datasets in check?

When I start an exploratory analysis on a large data set (many samples, many variables), I often find myself with hundreds of derived variables, and tonnes of different plots, and no real way to keep track of what's going where. Code ends up like spaghetti, because there's no direction from the start...

Are there any recommended methods for keeping an exploratory analysis neat and tidy? In particular, how do you deal multiple branches of exploration (including the ones that were dead-ends), and with different versions of plots?

For reference, I'm working on geoscientific data (many variables over time, sometimes also over space). I usually work with Python or R, and store everything in git, and have been trying out the IPython Notebook as well. However, it would be good if answers were somewhat general and useful for people in all fields, with other types of (large?) data.

• I would imagine a lot of advice you get would be equally applicable to simulation studies designed to evaluate competing methods of estimation or prediction. Mar 24 '14 at 14:08
• Yeah, this answer is probably required reading too: stats.stackexchange.com/questions/2910/…. I was thinking that there might be more specific advice, but I suppose there probably isn't really. Apr 7 '14 at 6:57

I think that frequently, the tendency to feel like you've gone down a rabbit hole with exploratory analyses is due to losing sight of the substantive question(s) you're asking. I do it myself, occasionally, and then have to remind myself what my goal(s) are. For example, am I trying to build a specific model, or evaluate the adequacy of an existing one? Am I looking for evidence of problems with the data (i.e., forensic data analysis)? Or, is this in the early stages of analysis, where I am investigating specific questions informally (e.g., is there a relationship between two variables?) before moving on to develop a formal model? In sum, if you catch yourself cranking out plots and tables but can't state clearly what your immediate goal is or why that plot/table is relevant, then you know you're getting pulled along by the activity (instead of being in control of it).

I try to approach exploratory data analysis like I do writing, whether that be writing a program or writing an article. In either case, I wouldn't start without making an outline first. That outline can change (and frequently does), of course, but to start writing without one is inefficient, and often yields a poor final product.

WRT organization, each analyst has to find a workflow that works for him or her—doing so is IMO more important than trying to follow rigidly someone else's workflow (though it is always helpful to get ideas from what others are doing). If you're working programmatically (i.e., writing code that can be run to generate/regenerate a set of results) and checking your work into git, then you're already miles ahead of many in this regard. I suspect that you may just need to spend some time organizing your code, and for that, I would suggest following your outline. For example, keep your analysis files relatively short and targeted, so that each answers one specific question (e.g., diagnostic plots for a specific regression model). Organize these into subdirectories at one or two levels, depending on the size and complexity of the project. In this way, the project becomes self-documenting; a list view of the directories, subdirectories and files (together with the comment at the top of each file) should, in theory, reproduce your outline.

Of course, in a large project, you might also have code that does data cleaning and management, code you've written to estimate a certain type of model, or other utilities you've written, and these won't fit within the substantive outline for your data analysis, so they should be organized in a different part of your project folder.

Update: After posting this, I realized that I didn't directly address your question about "dead ends." If you really decide that an entire set of analyses is of no value, then if you're working in git, you can always delete the corresponding file(s) with a commit message like "Abandoned this line of analysis because it wasn't productive." Unlike crumpling up what you've written and throwing it in the trash, you can always go back to what you did later on, if desired.

However, I think you'll find that if you proceed from an outline to which you've given some thought, you'll have fewer so-called dead-ends. Instead, if you spend time investigating a worthwhile and relevant question—even if this leads to a null finding or doesn't turn out like you anticipated—you probably still want to keep a record of what you've done and the outcome (at a minimum, so that you don't make the mistake of repeating this later on). Just move these to the bottom of your outline, in a sort of "Appendix."

I don't know how helpful a general answer will be. You're asking how to do something difficult; good answers will probably depend on the discipline and will probably be long and nuanced. :)

As far as organization goes, you're already using git, so next you should start using a makefile to execute the analysis. The makefile lays out how different files depend on each other (i.e., which statistics are derived from which code) and when you call make, everything that needs to be updated will.

Now, that doesn't help with the exploratory part. For EDA I use (mostly) R in emacs via ESS. You need need need a REPL for EDA. My workflow is to play with plots, estimates, etc. in ESS (in an exploratory.R type file), decide what I want to keep, then recode it so that it can be batch-executed by make. Re: git, I don't know how you're using it, but I use a single repository for each project (usually a single paper) and rebase the hell out of my codebase to keep a clean history; i.e. I use

$git merge meandering-branch --squash$ git add -p somefile
$git rebase -i master$ git reset HEAD --hard


way more than when I started with git, and way more than I'd recommend a beginner. If you're not familiar with all of those commands and options, you may want to learn more git. The biggest thing that's helped me is to be disciplined about making logically distinct commits; i.e. every commit should contain all of the changes that you might want to undo all at once in the future (and no more or less).

As far as actually exploring the data, I've found these books helpful and interesting, and they deal specifically with large datasets (at least in parts):

• The Graphics of Large Datasets, edited by Unwin, Theus, and Hofmann. via springerlink if you have access, otherwise individual chapters are probably available by googling.

• The handbook of data visualization, edited by Chen, Härdle, and Unwin. also via springerlink

• Data Analysis by Huber (2011)..

Two words: concept map. That's the only effective way I have found to divide and conquer large data sets or any concept that's really convoluted. http://en.wikipedia.org/wiki/Concept_maps

Personally, I think better on paper than on screen, so I just mind map what I'm dealing with before I even start to do any basic analysis. For a more professional diagram, there are lots of mind mapping software http://en.wikipedia.org/wiki/List_of_concept-_and_mind-mapping_software

• tells me what I have in terms of "core" variables and derived variables (if any)
• allows for the organization/formulation of a model based on theory/logic
• points to what variables I may be missing and/or could add if the relationships between the core variables don't pan out like I think they should

Edit:

As an example, here is the concept map for factor analysis: http://www.metacademy.org/graphs/concepts/factor_analysis#focus=factor_analysis&mode=explore Now this is purely for learning the concept, not performing analysis, but the idea is the same: to map out ahead of time what it makes sense to do, and then do it.

If you're looking for an automated/coded version of this, I don't think one exists. You can't automate the concept of modeling when you're trying to understand a system. (And that's a good thing because it would put plenty of people out of a job.)

• Hrm... This could do with a more detailed example. I have trouble seeing how this would help deal with the complexity I'm talking about. In particular, it doesn't help deal with what to do with analyses (derived data, plots, etc.) from investigation paths that lead to dead ends. Mar 19 '14 at 2:26
• The concept map is designed to only investigate paths that should lead somewhere based on subject-matter specific theory. If it turns out that a particular investigation didn't go anywhere, you make a note it on the concept map because that's your guide/to-do list.From there, you see immediately what derived variables are affected and what other investigations you could try. Mar 19 '14 at 2:37

You're already using git: why not use version control to organize your exploration? Create a new branch for each new "branch" of your exploration, and fork off branches for different versions of plots as well. This method will make it slightly more difficult to combine your end results, but you could always maintain an untracked directory where you could drop in the "gems" of your analysis. You'd probably want to somehow label your files in this directory to indicate which fork/commit they came from. This method has the added benefit of making it really easy to contrast different analyses via the diff command.

I would look into Business Intelligence tools... where similar issues arise. In particular (data warehouses, dimensional analysis,) hierarchies and drill downs.

The basic idea is that you try to represent your underlying data as aggregatable quantities ( counts, earnings etc rather than eg percentages). Then you design hierarchies to aggregate over the details ( eg months/ weeks/...) . This allows you to have simple overviews of all your data and then zoom in on particular areas. see eg http://cubes.databrewery.org/ (python) or excel power pivot