Data "exploration" vs data "snooping"/"torturing"?

Question

Many times I have come across informal warnings against "data snooping" (here's one amusing example), and I think I have an intuitive idea of roughly what that means, and why it may be a problem.

On the other hand, "exploratory data analysis" seems to be a perfectly respectable procedure in statistics, at least judging by the fact that a book with that title is still reverentially cited as a classic.

In my line of work I often come across what looks to me like rampant "data snooping", or perhaps it would be better described as "data torture", though those doing it seem to see the same activity as entirely reasonable and unproblematic "exploration".

Here's the typical scenario: costly experiment gets carried out (without much thought given to the subsequent analysis), the original researchers cannot readily discern a "story" in the gathered data, someone gets brought in to apply some "statistical wizardry", and who, after slicing and dicing the data every which way, finally manages to extract some publishable "story" from it.

Of course, there's usually some "validation" thrown in the final report/paper to show that the statistical analysis is on the up-and-up, but the blatant publish-at-all-cost attitude behind it all leaves me doubtful.

Unfortunately, my limited understanding of the do's and don'ts of data analysis keeps me from going beyond such vague doubts, so my conservative response is to basically disregard such findings.

My hope is that not only a better understanding of the distinction between exploration and snooping/torturing, but also, and more importantly, a better grasp of principles and techniques for detecting when that line has been crossed, will allow me to evaluate such findings in a way that can reasonably accounts for a less-than-optimal analytic procedure, and thus be able to go beyond my current rather simple-minded response of blanket disbelief.

---

EDIT: Thank you all for the very interesting comments and answers. Judging by their content, I think I may have not explained my question well enough. I hope this update will clarify matters.

My question here concerns not so much what I should do to avoid torturing my data (although this is a question that also interests me), but rather: how should I regard (or evaluate) results that I know for a fact have been arrived through such "data torture."

The situation gets more interesting in those (much rarer) cases in which, in addition, I am in the position to voice an opinion on such "findings" before they get submitted for publication.

At this point the most I can do is say something like "I don't know how much credence I can give to these findings, given what I know about the assumptions and procedures that went into getting them." This is too vague to be worth even saying. Wanting to go beyond such vagueness was the motivation for my post.

To be fair, my doubts here are based on more than seemingly questionable statistical methods. In fact, I see the latter more as consequence of the deeper problem: a combination of a cavalier attitude towards experimental design coupled with a categorical commitment to publishing the results as they stand (i.e. without any further experiments). Of course, follow-up projects are always envisioned, but it is simply out-of-the-question that not a single paper will come out of, say, "a refrigerator filled with 100,000 samples."

Statistics comes into the picture only as a means towards fulfilling this supreme objective. The only justification for latching onto the statistics (secondary as they are in the whole scenario) is that a frontal challenge to the assumption of "publication-at-all-cost" is simply pointless.

In fact, I can think of only one effective response in such situations: to propose some statistical test (not requiring additional experimentation) that truly tests the quality of the analysis. But I just don't have the chops in statistics for it. My hope (naive in retrospect) was to find out what I could study that may enable me to come up with such tests...

As I write this it dawns on me that, if it doesn't already exist, the world could use one new sub-branch of statistics, devoted to techniques for detecting and exposing "data-torture". (Of course, I don't mean getting carried away by the "torture" metaphor: the issue is not "data-torture" per-se, but the spurious "findings" it can lead to .)

Answer 1

There is a distinction which sometimes doesn't get enough attention, namely hypothesis generation vs. hypothesis testing, or exploratory analysis vs. hypothesis testing. You are allowed all the dirty tricks in the world to come up with your idea / hypothesis. But when you later test it, you must ruthlessly kill your darlings.

I'm a biologist working with high throughput data all the time, and yes, I do this "slicing and dicing" quite often. Most of the cases the experiment performed was not carefully designed; or maybe those who planned it did not account for all possible results. Or the general attitude when planning was "let's see what's in there". We end up with expensive, valuable and in themselves interesting data sets that I then turn around and around to come up with a story.

But then, it is only a story (possible bedtime). After you have selected a couple of interesting angles -- and here is the crucial point -- you must test it not only with independent data sets or independent samples, but preferably with an independent approach, an independent experimental system.

The importance of this last thing -- an independent experimental setup, not only independent set of measurements or samples -- is often underestimated. However, when we test 30,000 variables for significant difference, it often happens that while similar (but different) samples from the same cohort and analysed with the same method will not reject the hypothesis we based on the previous set. But then we turn to another type of experiment and another cohort, and our findings turn out to be the result of a methodological bias or are limited in their applicability.

That is why we often need several papers by several independent researchers to really accept a hypothesis or a model.

So I think such data torturing is fine, as long as you keep this distinction in mind and remember what you are doing, at what stage of scientific process you are. You can use moon phases or redefine 2+2 as long as you have an independent validation of the data. To put it on a picture:

Unfortunately, there are those who order a microarray to round up a paper after several experiments have been done and no story emerged, with the hope that the high throughput analysis shows something. Or they are confused about the whole hypothesis testing vs. generation thing.

Answer 2

Herman Friedman, my favorite professor in grad school, used to say that

"if you're not surprised, you haven't learned anything"

Strict avoidance of anything except the most rigorous testing of a priori defined hypotheses severely limits your ability to be surprised.

I think the key thing is that we are honest about what we are doing. If we are in a highly exploratory mode, we should say so. At the opposite end, one professor I know of told her student to change her hypotheses since the original ones were not found to be significant.

Answer 3

Let me add a few points:

• first of all, hypothesis generation is an important part of science. And non-predictive (exploratory/descriptive) results can be published.

• IMHO the trouble is not per se that data exploration is used on a data set and only parts of those findings are published. The problems are

  • not describing how much has been tried out
  • then drawing conclusions as if the study were a validation study for some predictive model / a hypothesis testing study

• Science and method development are iterative processes in a far more general way than just hypothesis generation - testing - generating new hypotheses - testing .... IMHO it is a matter of professional judgment what kind of proper conduct is necessary at what stage (see example below).

What I do:

• try to make people aware of the optimistic bias that results
  When I have a chance, I also show people how much of a difference that makes (feasible mostly with a lower level of the same problem, e.g. compare patient-independently validated data with internal performance estimates of hyper-parameter optimization routines, such as grid search for SVM paraters, "combined models" such as PCA-LDA, and so on. Not really feasible for the real data dredging, because so far, noone gave me the money to make a true replicate of a sensible sized study...)
• for papers that I'm coauthor of: insist on a discussion of the limitations of the conclusions. Make sure the conclusions are not formulated in a more general way than the study allows.
• Encourage co-workers to use their expert knowledge about the subject of the study and the process of data generation to decide how to treat the data instead of performing costly (in terms of the sample size you'd need to do that properly) optimization of model-"hyper"-parameters (such as what kind of pre-processing to use).
• in parallel: try to make people aware of how costly this optimization business is if done properly (whether this is called exploration or not is irrelevant, if done wrongly, it will have similar results like data dredging), e.g. Beleites, C. and Neugebauer, U. and Bocklitz, T. and Krafft, C. and Popp, J.: Sample size planning for classification models. Anal Chim Acta, 2013, 760, 25-33. DOI: 10.1016/j.aca.2012.11.007
  accepted manuscript on arXiv: 1211.1323

• Here's a study that finds this blind trying around also is often futile, e.g.
  J. Engel, J. Gerretzen, E. Szymańska, J. J. Jansen, G. Downey, L. Blanchet, L.M.C. Buydens: Breaking with trends in pre-processing?, TrAC Trends in Analytical Chemistry, 2013, 50, 96-106. DOI: 10.1016/j.trac.2013.04.015
  (they tried a large number of combinations of pre-processing steps and found that very few lead to better models than no pre-processing at all)

• Emphasise that I'm not torturing my data more than necessary:
  example:

  All preprocessing was decided exclusively using spectroscopic knowledge, and no data-driven preprocessing was performed.

  A follow-up paper using the same data as example for (different) theory development reads

  All pre-processing was decided by spectroscopic knowledge, no data-driven steps were included and no parameter optimization was performed. However, we checked that a PLS projection [45] of the spectra onto 25 latent variables as pre-processing for LR training did not lead to more than slight changes in the prediction (see supplementary figure S.2).

  Because meanwhile I was explicitly asked (on a conference by an editor of the journal CILS) to compare the models with PLS pre-processing.

• Take a practical point of view: E.g. in the astrocytoma study linked above, of course I still decided some points after looking at the data (such as what intensity threshold corresponds to measurements taken from outside the sample - which were then discarded). Other decisions I know to be uncritical (linear vs. quadratic baseline: my experience with that type of data suggests that this actually doesn't change much - which is also in perfect agreement with what Jasper Engel found on different data of similar type, so I wouldn't expect a large bias to come from deciding the type of baseline by looking at the data (the paper gives an argument why that is sensible).
  Based on the study we did, we can now say what should be tackled next and what should be changed. And because we are still in a comparatively early step of method development (looking at ex-vivo samples), it is not worth while to go through all the "homework" that will ultimately be needed before the method could be used in-vivo. E.g. at the present stage of the astrocytoma grading, resampling validation is a more sensible choice than external test set. I still emphasize that a truly external validation study will be needed at some point, because some performance characteristics can only be measured that way (e.g. the effects of instrument drift/proving that we can correct for these). But right now while we're still playing with ex-vivo samples and are solving other parts of the large problem (in the linked papers: how to deal with borderline cases), the gain in useful knowledge from a proper ex-vivo validation study is too low to be worth while the effort (IMHO: unless that were done in order to measure the bias due to data dredging).

• I once read an argument about statistical and reporting standards, and whether such should be decided to be necessary for a journal (don't remember which one) which convinced me: the idea expressed there was that there is no need for the editors to try agree on and enforce some standard (which will cause much futile discussion) because:

  • who uses the proper techniques is usually very aware/proud of that and will (and should) therefore report in detail what was done.
  • If a certain point (e.g. data dredging, validation not independent on patient level) is not clearly spelled out, the default assumption for reviewers/readers is that the study didn't adhere to the proper principles in that question (possibly because they didn't know better)

Answer 4

Sometimes the things you see as "data torture" aren't really. It's not always clear beforehand exactly what you're going to do with data to give what you believe are the genuine results of the experiment until you see it.

For example, with reaction time data for a decision task, you often want to reject times that aren't about the decision (i.e., when they are going so fast they are obviously just guessing and not making a decision). You can plot accuracy of the decision against RT to see where the guessing is generally occurring. But until you've tested that particular paradigm you have no way of knowing where the cutoffs are (in time, not accuracy). To some observers such a procedure looks like torturing the data but as long as it doesn't have anything directly to do with the hypothesis tests (you're not adjusting it based on tests) then it's not torturing the data.

Data snooping during an experiment is ok as long as it's done the right way. It's probably unethical to stick your experiment in a black box and only do the analysis when the planned number of subjects have been run. Sometimes it's hard to tell that there are issues with the experiment until you look at data and you should look at some as soon as possible. Data peeking is strongly disparaged because it's equated to seeing if p < 0.05 and deciding to continue. But there are lots of criteria by which you can decide to continue collecting that do not do anything harmful to your error rates.

Say you want to make sure that your variance estimate is within a known likely range. Small samples can have pretty far out variance estimates so you collect extra data until you know the sample is more representative. In the following simulation I expect the variance in each condition to be 1. I'm going to do something really crazy and sample each group independently for 10 samples and then add subjects until variance is close to 1.

Y <- replicate(1000, {
    y1 <- rnorm(10)
    while(var(y1) < 0.9 | var(y1) > 1.1) y1 <- c(y1, rnorm(1))
    y2 <- rnorm(10)
    while(var(y2) < 0.9 | var(y2) > 1.1) y2 <- c(y2, rnorm(1))
    c( t.test(y1, y2, var.equal = TRUE)$p.value, length(y1), length(y2) )
    })
range(Y[2,]) #range of N's in group 1
[1]   10 1173
range(Y[3,]) #range of N's in group 2
[1]   10 1283
sum(Y[1,] < 0.05) / ncol(Y)
[1] 0.045

So, I've just gone bonkers with the sampling and making my variances close to expected and I still don't affect alpha much (it's a little under 0.05). A few more constraints like the N's must be equal in each group and can't be more than 30 and alpha is pretty much right on 0.05. But what about SE? What if I instead tried to make the SE a given value? That's actually a really interesting idea because I'm in turn setting the width of CI in advance (but not the location).

se <- function(x) sqrt(var(x) / length(x))
Y <- replicate(1000, {
        y1 <- rnorm(10)
        y2 <- rnorm(10)
        while(se(y1) > 0.2 | se(y2) > 0.2) {
            y1 <- c(y1, rnorm(1)); y2 <- c(y2, rnorm(1))
        }
        c( t.test(y1, y2, var.equal = TRUE)$p.value, length(y1) )
        })
range(Y[2,]) #range of N's in group 1 and 2 (they're equal now)
[1] 10 46
sum(Y[1,] < 0.05) / ncol(Y)
[1] 0.053

Again, alpha changed a small amount even though I've allowed N's to roam up to 46 from the original 10 based on data snooping. More importantly, the SE's all fall in a narrow range in each of the experiments. It's easy to make a small alpha adjustment to fix that if it's a concern. The point is that some data snooping does little to no harm and can even bring benefits.

(BTW, what I'm showing isn't some magic bullet. You don't actually reduce the number of subjects in the long run doing this because power for the varying N's simulation is about the same as for a simulation of the average N's)

None of the above contradicts the recent literature on adding subjects after an experiment started. In those studies they looked at simulations where you added subjects after doing a hypothesis test in order to get the p-value lower. That's still bad and can extraordinarily inflate alpha. Furthermore, I really like January and Peter Flom's answers. I just wanted to point out that looking at data while you're collecting it, and even changing a planned N while collecting, are not necessarily bad things.

Answer 5

This is really a cultural problem of unbalanced thinking, where publication bias leads to the favouring of positive results and our competitive nature requires editors and researchers to be seen to be producing results of interest that are novel or contentious, for example, in the sense of rebutting someone else's results. In medical research there is considerable progress being made to redress this problem by the compulsory registration of trials and publication of results with records of abandoned trials to also be made public. I understand that since publication in journals for unsuccessful research may not be practicable, there are plans to keep a publicly available database of them. Unusual results that can not be replicated are not necessarily a result of misdemeanour, as with perhaps 50,000 (a guess) researchers worldwide doing several experiments a year, some pretty unusual results are to be expected from time to time.

Using different methods is not necessarily a solution either. For example, what chemist would mix reagents in different ways in different conditions and expect the same results as a matter of course?