23
$\begingroup$

I was reading this article in Nature in which some fallacies are explained in the context of data analysis. I noticed that the Texas sharpshooter fallacy was particularly difficult to avoid:

A cognitive trap that awaits during data analysis is illustrated by the fable of the Texas sharpshooter: an inept marksman who fires a random pattern of bullets at the side of a barn, draws a target around the biggest clump of bullet holes, and points proudly at his success.

His bullseye is obviously laughable — but the fallacy is not so obvious to gamblers who believe in a 'hot hand' when they have a streak of wins, or to people who see supernatural significance when a lottery draw comes up as all odd numbers.

Nor is it always obvious to researchers. “You just get some encouragement from the data and then think, well, this is the path to go down,” says Pashler. “You don't realize you had 27 different options and you picked the one that gave you the most agreeable or interesting results, and now you're engaged in something that's not at all an unbiased representation of the data.”

I think that kind of exploration work is commonplace and often, hypotheses are constructed based on that part of the analysis. There is a whole approach (EDA) dedicated to this process:

Exploratory data analysis was promoted by John Tukey to encourage statisticians to explore the data, and possibly formulate hypotheses that could lead to new data collection and experiments

It looks like any exploratory process performed without having a hypothesis beforehand is prone to generate spurious hypotheses.

Notice that the description of EDA above actually talks about new data collection and experiments. I understand that after new data have been collected, then a confirmatory data analysis (CDA) is appropriate. However, I don't think this distinction is made very clearly, and although a separation of EDA and CDA would be ideal, surely there are some circumstances in which this is not feasible. I would go as far as to say that following this separation strictly is uncommon and most practitioners don't subscribe to the EDA paradigm at all.

So my question is: Does EDA (or any informal process of exploring data) make it more likely to fall for the Texas sharpshooter fallacy?

Improve this question
$\endgroup$
4
  • 3
    $\begingroup$ I don't know precisely what you mean by "spurious hypothesis". The spirit of exploratory data analysis is to look at the data and be open to a variety of patterns, including patterns you didn't expect. No less, and not much more. Nothing in exploratory data analysis guarantees good ideas and nothing rules out or lets you dispense with thinking critically or linking what you do to the underlying science (wide sense). So, there is some risk here of criticising EDA for things nobody experienced denies. or no one supports. $\endgroup$
    – Nick Cox
    Oct 16, 2015 at 9:05
  • 3
    $\begingroup$ What's hardest to learn and teach in EDA is precisely what significance tests are supposed to help on (in the most optimistic accounts): to get in the habit of not over-interpreting details in the data that aren't important enough to be worth attention. I would argue that many accounts of EDA don't push hard enough the idea that for a pattern to be taken seriously it must be identifiable in different datasets, but that neglect is common across statistical science. $\endgroup$
    – Nick Cox
    Oct 16, 2015 at 10:08
  • 1
    $\begingroup$ Thanks. The issue is that generating many hypotheses and test them in the same dataset is really dangerous because you are likely to confirm one of them even if it is false. As Creosote describes, a correction in p-values would be needed. Unfortunately, I have never seen this done in practice. $\endgroup$
    – r_31415
    Oct 16, 2015 at 14:16
  • 2
    $\begingroup$ From learning (French) EDA in the early 1980's, I got the impression that it was actually much easier to bend your analysis towards the intended conclusion(s) than with a stronger statistical structure... $\endgroup$
    – Xi'an
    Nov 7, 2015 at 8:42

6 Answers 6

Reset to default
12
+50
$\begingroup$

If one views the role of EDA strictly as generating hypotheses, then no the sharpshooter fallacy does not apply. However, it is very important that subsequent confirmatory trials are indeed independent. Many researchers attempt to "reconcile differences" with things like pooled analyses, meta analyses, and Bayesian methods. This means that at least some of the evidence presented in such an analysis includes "the circle around the random bullet holes".

Improve this answer
$\endgroup$
1
  • 5
    $\begingroup$ Exactly. The problem with so much exploratory data analysis is that the same set is used for both training (identifying where the bullets landed) and test (drawing the circle around them). $\endgroup$
    – Michael K
    Nov 6, 2015 at 20:57
11
$\begingroup$

This paints a very negative view of exploratory data analysis. While the argument is not wrong, it's really saying "what can go wrong when I use a very important tool in the wrong manner?"

Accepting unadjusted p-values from EDA methods will lead to vastly inflated type I error rates. But I think Tukey would not be happy with anyone doing this. The point of EDA is not to make definitive conclusions about relations in the data, but rather to look for potential novel relations in the data to follow up on.

Leaving out this step in the larger scientific process is essentially hamstringing science to never be able to find new interesting aspects of our data, outside of pure logical deduction. Ever try to logically deduce how over expression of a set of genes will affect survival of a cell? Hint: it's not very easy (one of our favorite jokes among the bioinformatics staff at my work was when a physicist asked "Why don't you just simulate the physical properties of different gene interactions? It's a finite parameters space.")

Personally, I think confusion about this can lead to a great slow down in scientific progress. I know too many non-statistical researchers that will state that they do not want to do EDA procedures on preliminary data, because they "know that EDA can be bad".

In conclusion, it's absolutely true that using EDA methods and treating them as confirmatory data analysis methods will lead to invalid results. However, the lack of proper use of EDA can lead to almost no results.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you. I wouldn't worry too much about having few people engaging in some kind of exploratory analysis. I think the opposite is true; many people are doing that exploratory work but probably without taking adequate precautions to prevent type I errors as you described. However, it is interesting that you know people that have a negative opinion on EDA. If they don't want to do it in preliminary data, then when do they feel comfortable doing EDA (or EDA-like) work? $\endgroup$
    – r_31415
    Nov 4, 2015 at 20:11
  • $\begingroup$ My experience is that non-statistical researchers are used to hearing "multiple comparisons are problematic" and so when they come to me with data, they are eager to say that they want to avoid making multiple comparisons, even with preliminary data. Of course, a more complete understanding of the problem would be that you want to avoid multiple comparisons in a CDA study. $\endgroup$
    – Cliff AB
    Nov 5, 2015 at 1:27
  • $\begingroup$ I understand. That makes more sense. $\endgroup$
    – r_31415
    Nov 5, 2015 at 21:05
5
$\begingroup$

It looks like any exploratory process performed without having a hypothesis beforehand is prone to generate spurious hypotheses.

I would temper this statement and express it a little differently: Choosing a hypothesis to test based on the data undermines the test if one doesn't use the correct null hypothesis. The thrust of the Nature article is, essentially, that it's easy for analysts to kid themselves into ignoring all of the multiple comparisons they're implicitly making during exploration.

Nature quotes Andrew Gelman, but doesn't mention his paper with Eric Loken about just this topic. An excerpt:

When criticisms of multiple comparisons have come up in regards to some of the papers we discuss here, the researchers never respond that they had chosen all the details of their data processing and data analysis ahead of time; rather, they claim that they picked only one analysis for the particular data they saw. Intuitive as this defense may seem, it does not address the fundamental frequentist concern of multiple comparisons.

Another:

It’s not that the researchers performed hundreds of different comparisons and picked ones that were statistically significant. Rather, they start with a somewhat-formed idea in their mind of what comparison to perform, and they refine that idea in light of the data. They saw a pattern in red and pink, and they combined the colors.

Succinctly:

There is a one-to-many mapping from scientific to statistical hypotheses.

And one more, emphasis mine:

In all the cases we have discussed, the published analysis has a story that is consistent with the scientific hypotheses that motivated the work, but other data patterns (which, given the sample sizes, could easily have occurred by chance) would naturally have led to different data analyses (for example, a focus on main effects rather than interactions, or a different choice of data subsets to compare) which equally could have been used to support the research hypotheses. The result remains, as we have written elsewhere, a sort of machine for producing and publicizing random patterns.

In short, it's not that EDA leads to a "spurious hypothesis"; it's that testing a hypothesis with the same dataset that prompted the hypothesis can lead to spurious conclusions.

If you're interested in conquering this obstacle, Gelman has another paper arguing that many of these problems disappear in a Bayesian framework, and the paper with Loken references "pre-publication replication" as anecdotally described in the first section of this paper.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you. Very interesting. I will take a look at Gelman's paper on multiple comparisons. $\endgroup$
    – r_31415
    Nov 7, 2015 at 5:03
3
$\begingroup$

Almost by definition, yes, of course EDA without CDA attracts Texas sharpshooters.

The difficulty when CDA is not possible (perhaps no further data can be obtained) is in being honest with yourself about how many tests you've really performed, and thus in assigning some kind of $p$-value to your discovery. Even in cases when the search space could in principle be counted, the $p$-value calculation is either done wrongly or not at all: see wikipedia for a notorious example.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you. Yes, a correction would be required. I don't think taking this into account is very common. $\endgroup$
    – r_31415
    Oct 16, 2015 at 14:24
3
$\begingroup$

Just to add to the already great answers: There is a middle ground between a full CDA and just accepting your EDA results at face value. Once you've found a possible feature of interest (or hypothesis), you can get a sense of its robustness by performing cross-validation (CV) or bootstrap simulations. If your findings depend on only a few key observations, then CV or Bootstrap will show that many of the folds(CV) or boostrap samples fail to reproduce the observed feature.

This is not a foolproof method, but its a good intermediate check before going for a full CDA (or purposefully holding out a "validation set" from your initial data pool).

Improve this answer
$\endgroup$
0
$\begingroup$

The most rigorous criterion for data model selection is the degree to which is approximates the Kolmogorov Complexity of the data -- which is to say the degree to which it losslessly compress the data. This can, in theory, result from exploratory data analysis alone.

See "Causal deconvolution by algorithmic generative models"

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.