32
$\begingroup$

Many times I have come across informal warnings against "data snooping" (here's one amusing example), and I think I have an intuitive idea of roughly what that means, and why it may be a problem.

On the other hand, "exploratory data analysis" seems to be a perfectly respectable procedure in statistics, at least judging by the fact that a book with that title is still reverentially cited as a classic.

In my line of work I often come across what looks to me like rampant "data snooping", or perhaps it would be better described as "data torture", though those doing it seem to see the same activity as entirely reasonable and unproblematic "exploration".

Here's the typical scenario: costly experiment gets carried out (without much thought given to the subsequent analysis), the original researchers cannot readily discern a "story" in the gathered data, someone gets brought in to apply some "statistical wizardry", and who, after slicing and dicing the data every which way, finally manages to extract some publishable "story" from it.

Of course, there's usually some "validation" thrown in the final report/paper to show that the statistical analysis is on the up-and-up, but the blatant publish-at-all-cost attitude behind it all leaves me doubtful.

Unfortunately, my limited understanding of the do's and don'ts of data analysis keeps me from going beyond such vague doubts, so my conservative response is to basically disregard such findings.

My hope is that not only a better understanding of the distinction between exploration and snooping/torturing, but also, and more importantly, a better grasp of principles and techniques for detecting when that line has been crossed, will allow me to evaluate such findings in a way that can reasonably accounts for a less-than-optimal analytic procedure, and thus be able to go beyond my current rather simple-minded response of blanket disbelief.


EDIT: Thank you all for the very interesting comments and answers. Judging by their content, I think I may have not explained my question well enough. I hope this update will clarify matters.

My question here concerns not so much what I should do to avoid torturing my data (although this is a question that also interests me), but rather: how should I regard (or evaluate) results that I know for a fact have been arrived through such "data torture."

The situation gets more interesting in those (much rarer) cases in which, in addition, I am in the position to voice an opinion on such "findings" before they get submitted for publication.

At this point the most I can do is say something like "I don't know how much credence I can give to these findings, given what I know about the assumptions and procedures that went into getting them." This is too vague to be worth even saying. Wanting to go beyond such vagueness was the motivation for my post.

To be fair, my doubts here are based on more than seemingly questionable statistical methods. In fact, I see the latter more as consequence of the deeper problem: a combination of a cavalier attitude towards experimental design coupled with a categorical commitment to publishing the results as they stand (i.e. without any further experiments). Of course, follow-up projects are always envisioned, but it is simply out-of-the-question that not a single paper will come out of, say, "a refrigerator filled with 100,000 samples."

Statistics comes into the picture only as a means towards fulfilling this supreme objective. The only justification for latching onto the statistics (secondary as they are in the whole scenario) is that a frontal challenge to the assumption of "publication-at-all-cost" is simply pointless.

In fact, I can think of only one effective response in such situations: to propose some statistical test (not requiring additional experimentation) that truly tests the quality of the analysis. But I just don't have the chops in statistics for it. My hope (naive in retrospect) was to find out what I could study that may enable me to come up with such tests...

As I write this it dawns on me that, if it doesn't already exist, the world could use one new sub-branch of statistics, devoted to techniques for detecting and exposing "data-torture". (Of course, I don't mean getting carried away by the "torture" metaphor: the issue is not "data-torture" per-se, but the spurious "findings" it can lead to .)

$\endgroup$
  • 1
    $\begingroup$ @BabakP That quotation appears in six answers here, including in the stats jokes and stats quotations threads. (The latter is a good source for relevant quotations if you are ever hunting some down.) $\endgroup$ – whuber Sep 16 '13 at 16:13
  • 7
    $\begingroup$ I don't think there is any distinction between the techniques used in 'data snooping' & in 'exploratory data analysis' - the derogatory use of the former term is for an exploratory analysis misleadingly presented as a confirmatory analysis. $\endgroup$ – Scortchi - Reinstate Monica Sep 16 '13 at 16:14
  • 8
    $\begingroup$ Feynman, in the book you reference, already answers this question: "If he wants to test this hypothesis [found through exploration], ... he must do another experiment." What you seem to be asking concerns whether Feynman may have been too extreme ("exaggerating a little"): to what extent, if at all, can formal testing of hypotheses be justified when they were developed by exploring the same data? $\endgroup$ – whuber Sep 16 '13 at 16:18
  • 2
    $\begingroup$ @whuber: in practice it is even more dramatic, because often testing with different data, but the same experimental setup or type of experiment will inadvertently lead to similar results. $\endgroup$ – January Sep 16 '13 at 16:24
  • 1
    $\begingroup$ @January: that depends on your data/experiments I think. Consider e.g. biological/medical research. For the data I see, the largest variation is usually between patients (subjects). Repeating the experiment with new patients will hopefully lead to similar results, but in practice this is quite often not the case (i.e. prediction results of models developed on the first set of patients are much worse than expected, which means that overfitting occured, so the data in the first experiment was "tortured") $\endgroup$ – cbeleites unhappy with SX Sep 17 '13 at 19:16
22
$\begingroup$

There is a distinction which sometimes doesn't get enough attention, namely hypothesis generation vs. hypothesis testing, or exploratory analysis vs. hypothesis testing. You are allowed all the dirty tricks in the world to come up with your idea / hypothesis. But when you later test it, you must ruthlessly kill your darlings.

I'm a biologist working with high throughput data all the time, and yes, I do this "slicing and dicing" quite often. Most of the cases the experiment performed was not carefully designed; or maybe those who planned it did not account for all possible results. Or the general attitude when planning was "let's see what's in there". We end up with expensive, valuable and in themselves interesting data sets that I then turn around and around to come up with a story.

But then, it is only a story (possible bedtime). After you have selected a couple of interesting angles -- and here is the crucial point -- you must test it not only with independent data sets or independent samples, but preferably with an independent approach, an independent experimental system.

The importance of this last thing -- an independent experimental setup, not only independent set of measurements or samples -- is often underestimated. However, when we test 30,000 variables for significant difference, it often happens that while similar (but different) samples from the same cohort and analysed with the same method will not reject the hypothesis we based on the previous set. But then we turn to another type of experiment and another cohort, and our findings turn out to be the result of a methodological bias or are limited in their applicability.

That is why we often need several papers by several independent researchers to really accept a hypothesis or a model.

So I think such data torturing is fine, as long as you keep this distinction in mind and remember what you are doing, at what stage of scientific process you are. You can use moon phases or redefine 2+2 as long as you have an independent validation of the data. To put it on a picture:

enter image description here

Unfortunately, there are those who order a microarray to round up a paper after several experiments have been done and no story emerged, with the hope that the high throughput analysis shows something. Or they are confused about the whole hypothesis testing vs. generation thing.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I suppose that one could construe what I've seen as "hypothesis generation", but the aim of the manipulations I'm talking about is most definitely to publish the results obtained from the "tortured" data, and to do so in the highest-impact journal that will accept the paper. Needless to say, such papers never carry any suggestion of the tortured origins of their findings. In fact, AFAICT, the authors are not at all troubled by this. And yet, I think a majority of such papers' readers would heavily discount the findings if they knew exactly how much data-torture went into getting them... $\endgroup$ – kjo Sep 16 '13 at 18:48
  • 1
    $\begingroup$ @kjo: hypothesis generation is a part of the scientific process that definitively can be published. So that's no reason. $\endgroup$ – cbeleites unhappy with SX Sep 18 '13 at 13:27
  • $\begingroup$ @January: you forgot to mention the DoE "take all samples we can get - they'll anyways be too few" - which is the most frequent DoE I encounter. $\endgroup$ – cbeleites unhappy with SX Sep 18 '13 at 13:29
  • $\begingroup$ @cbeleites: well, I wouldn't dream of criticizing this attitude in general; usually the experiments could benefit from a larger number of replicates. But I agree that often experimentalists tend to include as many conditions (sample types, strains, variants, classes etc) as only physically possible, making analysis a nightmare and sometimes totally obscuring the question. $\endgroup$ – January Sep 18 '13 at 15:55
13
$\begingroup$

Herman Friedman, my favorite professor in grad school, used to say that

"if you're not surprised, you haven't learned anything"

Strict avoidance of anything except the most rigorous testing of a priori defined hypotheses severely limits your ability to be surprised.

I think the key thing is that we are honest about what we are doing. If we are in a highly exploratory mode, we should say so. At the opposite end, one professor I know of told her student to change her hypotheses since the original ones were not found to be significant.

| cite | improve this answer | |
$\endgroup$
  • 4
    $\begingroup$ Nothing wrong with rigorously testing a priori defined hypotheses and snooping the same data to suggest the next a priori defined hypotheses to be rigorously tested. And if we're in an even slightly exploratory mode we should say so - just say what we really did - & let others decide with exactly how large a pinch of salt they want to take our results, however convinced of their validity we may be ourselves. I'd like to give this answer more than one vote for emphasizing honesty. $\endgroup$ – Scortchi - Reinstate Monica Sep 16 '13 at 22:25
7
$\begingroup$

Let me add a few points:

  • first of all, hypothesis generation is an important part of science. And non-predictive (exploratory/descriptive) results can be published.

  • IMHO the trouble is not per se that data exploration is used on a data set and only parts of those findings are published. The problems are

    • not describing how much has been tried out
    • then drawing conclusions as if the study were a validation study for some predictive model / a hypothesis testing study
  • Science and method development are iterative processes in a far more general way than just hypothesis generation - testing - generating new hypotheses - testing .... IMHO it is a matter of professional judgment what kind of proper conduct is necessary at what stage (see example below).

What I do:

  • try to make people aware of the optimistic bias that results
    When I have a chance, I also show people how much of a difference that makes (feasible mostly with a lower level of the same problem, e.g. compare patient-independently validated data with internal performance estimates of hyper-parameter optimization routines, such as grid search for SVM paraters, "combined models" such as PCA-LDA, and so on. Not really feasible for the real data dredging, because so far, noone gave me the money to make a true replicate of a sensible sized study...)
  • for papers that I'm coauthor of: insist on a discussion of the limitations of the conclusions. Make sure the conclusions are not formulated in a more general way than the study allows.
  • Encourage co-workers to use their expert knowledge about the subject of the study and the process of data generation to decide how to treat the data instead of performing costly (in terms of the sample size you'd need to do that properly) optimization of model-"hyper"-parameters (such as what kind of pre-processing to use).
  • in parallel: try to make people aware of how costly this optimization business is if done properly (whether this is called exploration or not is irrelevant, if done wrongly, it will have similar results like data dredging), e.g. Beleites, C. and Neugebauer, U. and Bocklitz, T. and Krafft, C. and Popp, J.: Sample size planning for classification models. Anal Chim Acta, 2013, 760, 25-33. DOI: 10.1016/j.aca.2012.11.007
    accepted manuscript on arXiv: 1211.1323
  • Here's a study that finds this blind trying around also is often futile, e.g.
    J. Engel, J. Gerretzen, E. Szymańska, J. J. Jansen, G. Downey, L. Blanchet, L.M.C. Buydens: Breaking with trends in pre-processing?, TrAC Trends in Analytical Chemistry, 2013, 50, 96-106. DOI: 10.1016/j.trac.2013.04.015
    (they tried a large number of combinations of pre-processing steps and found that very few lead to better models than no pre-processing at all)

  • Emphasise that I'm not torturing my data more than necessary:
    example:

    All preprocessing was decided exclusively using spectroscopic knowledge, and no data-driven preprocessing was performed.

    A follow-up paper using the same data as example for (different) theory development reads

    All pre-processing was decided by spectroscopic knowledge, no data-driven steps were included and no parameter optimization was performed. However, we checked that a PLS projection [45] of the spectra onto 25 latent variables as pre-processing for LR training did not lead to more than slight changes in the prediction (see supplementary figure S.2).

    Because meanwhile I was explicitly asked (on a conference by an editor of the journal CILS) to compare the models with PLS pre-processing.

  • Take a practical point of view: E.g. in the astrocytoma study linked above, of course I still decided some points after looking at the data (such as what intensity threshold corresponds to measurements taken from outside the sample - which were then discarded). Other decisions I know to be uncritical (linear vs. quadratic baseline: my experience with that type of data suggests that this actually doesn't change much - which is also in perfect agreement with what Jasper Engel found on different data of similar type, so I wouldn't expect a large bias to come from deciding the type of baseline by looking at the data (the paper gives an argument why that is sensible).
    Based on the study we did, we can now say what should be tackled next and what should be changed. And because we are still in a comparatively early step of method development (looking at ex-vivo samples), it is not worth while to go through all the "homework" that will ultimately be needed before the method could be used in-vivo. E.g. at the present stage of the astrocytoma grading, resampling validation is a more sensible choice than external test set. I still emphasize that a truly external validation study will be needed at some point, because some performance characteristics can only be measured that way (e.g. the effects of instrument drift/proving that we can correct for these). But right now while we're still playing with ex-vivo samples and are solving other parts of the large problem (in the linked papers: how to deal with borderline cases), the gain in useful knowledge from a proper ex-vivo validation study is too low to be worth while the effort (IMHO: unless that were done in order to measure the bias due to data dredging).

  • I once read an argument about statistical and reporting standards, and whether such should be decided to be necessary for a journal (don't remember which one) which convinced me: the idea expressed there was that there is no need for the editors to try agree on and enforce some standard (which will cause much futile discussion) because:

    • who uses the proper techniques is usually very aware/proud of that and will (and should) therefore report in detail what was done.
    • If a certain point (e.g. data dredging, validation not independent on patient level) is not clearly spelled out, the default assumption for reviewers/readers is that the study didn't adhere to the proper principles in that question (possibly because they didn't know better)
| cite | improve this answer | |
$\endgroup$
4
$\begingroup$

Sometimes the things you see as "data torture" aren't really. It's not always clear beforehand exactly what you're going to do with data to give what you believe are the genuine results of the experiment until you see it.

For example, with reaction time data for a decision task, you often want to reject times that aren't about the decision (i.e., when they are going so fast they are obviously just guessing and not making a decision). You can plot accuracy of the decision against RT to see where the guessing is generally occurring. But until you've tested that particular paradigm you have no way of knowing where the cutoffs are (in time, not accuracy). To some observers such a procedure looks like torturing the data but as long as it doesn't have anything directly to do with the hypothesis tests (you're not adjusting it based on tests) then it's not torturing the data.

Data snooping during an experiment is ok as long as it's done the right way. It's probably unethical to stick your experiment in a black box and only do the analysis when the planned number of subjects have been run. Sometimes it's hard to tell that there are issues with the experiment until you look at data and you should look at some as soon as possible. Data peeking is strongly disparaged because it's equated to seeing if p < 0.05 and deciding to continue. But there are lots of criteria by which you can decide to continue collecting that do not do anything harmful to your error rates.

Say you want to make sure that your variance estimate is within a known likely range. Small samples can have pretty far out variance estimates so you collect extra data until you know the sample is more representative. In the following simulation I expect the variance in each condition to be 1. I'm going to do something really crazy and sample each group independently for 10 samples and then add subjects until variance is close to 1.

Y <- replicate(1000, {
    y1 <- rnorm(10)
    while(var(y1) < 0.9 | var(y1) > 1.1) y1 <- c(y1, rnorm(1))
    y2 <- rnorm(10)
    while(var(y2) < 0.9 | var(y2) > 1.1) y2 <- c(y2, rnorm(1))
    c( t.test(y1, y2, var.equal = TRUE)$p.value, length(y1), length(y2) )
    })
range(Y[2,]) #range of N's in group 1
[1]   10 1173
range(Y[3,]) #range of N's in group 2
[1]   10 1283
sum(Y[1,] < 0.05) / ncol(Y)
[1] 0.045

So, I've just gone bonkers with the sampling and making my variances close to expected and I still don't affect alpha much (it's a little under 0.05). A few more constraints like the N's must be equal in each group and can't be more than 30 and alpha is pretty much right on 0.05. But what about SE? What if I instead tried to make the SE a given value? That's actually a really interesting idea because I'm in turn setting the width of CI in advance (but not the location).

se <- function(x) sqrt(var(x) / length(x))
Y <- replicate(1000, {
        y1 <- rnorm(10)
        y2 <- rnorm(10)
        while(se(y1) > 0.2 | se(y2) > 0.2) {
            y1 <- c(y1, rnorm(1)); y2 <- c(y2, rnorm(1))
        }
        c( t.test(y1, y2, var.equal = TRUE)$p.value, length(y1) )
        })
range(Y[2,]) #range of N's in group 1 and 2 (they're equal now)
[1] 10 46
sum(Y[1,] < 0.05) / ncol(Y)
[1] 0.053

Again, alpha changed a small amount even though I've allowed N's to roam up to 46 from the original 10 based on data snooping. More importantly, the SE's all fall in a narrow range in each of the experiments. It's easy to make a small alpha adjustment to fix that if it's a concern. The point is that some data snooping does little to no harm and can even bring benefits.

(BTW, what I'm showing isn't some magic bullet. You don't actually reduce the number of subjects in the long run doing this because power for the varying N's simulation is about the same as for a simulation of the average N's)

None of the above contradicts the recent literature on adding subjects after an experiment started. In those studies they looked at simulations where you added subjects after doing a hypothesis test in order to get the p-value lower. That's still bad and can extraordinarily inflate alpha. Furthermore, I really like January and Peter Flom's answers. I just wanted to point out that looking at data while you're collecting it, and even changing a planned N while collecting, are not necessarily bad things.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ None of these things are 'fine' in the sense of not affecting the sampling distribution of your test statistics. Perfectly sensible responses to surprises of course (cf @Peter's answer), but they do dilute somewhat the confirmatory nature of your experiment, increasing the 'researcher degrees of freedom'. It's precisely to avoid surprises that we do pilot studies to fix the protocol, & define stopping rules beforehand, taking them into account in the analysis. The goal is a well-defined procedure that can be independently replicated to demonstrate the validity of your results. $\endgroup$ – Scortchi - Reinstate Monica Sep 16 '13 at 21:22
  • $\begingroup$ You can feel free to run the simulations yourself but having a variance based stopping rule (over a reasonable minimum N) will have no impact on alpha and will generate an expected power. You can even have an SE based stopping rule and get consistent SE's and those won't affect alpha, or beta. You just can't have a p based stopping rule. All of the criticisms of modifying N are about doing it after a hypothesis test (there should be other things included as well). There is the potential that this causes temptation...but I'm ignoring that. $\endgroup$ – John Sep 16 '13 at 23:18
  • $\begingroup$ As for the reaction time distribution, you're suggesting it's better to pick a fixed cut point based on a pilot rather than figuring out when each subject is guessing based on logistic regression and use their own cut point? (of course the accuracy cut point is fixed, just not the reaction time one). $\endgroup$ – John Sep 16 '13 at 23:28
  • $\begingroup$ (1) Variance-based stopping rule: It affects the variance estimate, & therefore can affect error rates when the experiment's analysed as if the sample size had been fixed beforehand. There's a tension between the caveat of "beyond a reasonable minimum N" given in your comment & the "small sample sizes" referred to in your answer; doubtless you have the statistical nous to know what approximations are good enough when, but not everyone does. More generally, an unimpeachable approach is to clearly define the stopping rule before the experiment. $\endgroup$ – Scortchi - Reinstate Monica Sep 17 '13 at 16:53
  • $\begingroup$ (2) Reaction time distribution: No (though I admittedly had something like that in mind); I was suggesting that whatever the method used to remove unreliable observations, it would be better developed from a pilot study, & then applied in a confirmatory experiment. $\endgroup$ – Scortchi - Reinstate Monica Sep 17 '13 at 16:54
0
$\begingroup$

This is really a cultural problem of unbalanced thinking, where publication bias leads to the favouring of positive results and our competitive nature requires editors and researchers to be seen to be producing results of interest that are novel or contentious, for example, in the sense of rebutting someone else's results. In medical research there is considerable progress being made to redress this problem by the compulsory registration of trials and publication of results with records of abandoned trials to also be made public. I understand that since publication in journals for unsuccessful research may not be practicable, there are plans to keep a publicly available database of them. Unusual results that can not be replicated are not necessarily a result of misdemeanour, as with perhaps 50,000 (a guess) researchers worldwide doing several experiments a year, some pretty unusual results are to be expected from time to time.

Using different methods is not necessarily a solution either. For example, what chemist would mix reagents in different ways in different conditions and expect the same results as a matter of course?

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.