Texas sharpshooter fallacy in exploratory data analysis I was reading this article in Nature in which some fallacies are explained in the context of data analysis. I noticed that the Texas sharpshooter fallacy was particularly difficult to avoid:

A cognitive trap that awaits during data analysis is illustrated by
  the fable of the Texas sharpshooter: an inept marksman who fires a
  random pattern of bullets at the side of a barn, draws a target around
  the biggest clump of bullet holes, and points proudly at his success.
His bullseye is obviously laughable — but the fallacy is not so
  obvious to gamblers who believe in a 'hot hand' when they have a
  streak of wins, or to people who see supernatural significance when a
  lottery draw comes up as all odd numbers.
Nor is it always obvious to researchers. “You just get some
  encouragement from the data and then think, well, this is the path to
  go down,” says Pashler. “You don't realize you had 27 different
  options and you picked the one that gave you the most agreeable or
  interesting results, and now you're engaged in something that's not at
  all an unbiased representation of the data.”

I think that kind of exploration work is commonplace and often, hypotheses are constructed based on that part of the analysis. There is a whole approach (EDA) dedicated to this process:

Exploratory data analysis was promoted by John Tukey to encourage
  statisticians to explore the data, and possibly formulate hypotheses
  that could lead to new data collection and experiments

It looks like any exploratory process performed without having a hypothesis beforehand is prone to generate spurious hypotheses. 
Notice that the description of EDA above actually talks about new data collection and experiments. I understand that after new data have been collected, then a confirmatory data analysis (CDA) is appropriate. However, I don't think this distinction is made very clearly, and although a separation of EDA and CDA would be ideal, surely there are some circumstances in which this is not feasible. I would go as far as to say that following this separation strictly is uncommon and most practitioners don't subscribe to the EDA paradigm at all. 
So my question is: Does EDA (or any informal process of exploring data) make it more likely to fall for the Texas sharpshooter fallacy?
 A: 
It looks like any exploratory process performed without having a hypothesis beforehand is prone to generate spurious hypotheses.

I would temper this statement and express it a little differently: Choosing a hypothesis to test based on the data undermines the test if one doesn't use the correct null hypothesis. The thrust of the Nature article is, essentially, that it's easy for analysts to kid themselves into ignoring all of the multiple comparisons they're implicitly making during exploration.
Nature quotes Andrew Gelman, but doesn't mention his paper with Eric Loken about just this topic. An excerpt:

When criticisms of multiple comparisons have come up in regards to some of the papers we discuss here, the researchers never respond that they had chosen all the details of their data processing and data analysis ahead of time; rather, they claim that they picked only one analysis for the particular data they saw. Intuitive as this defense may seem, it does not address the fundamental frequentist concern of multiple comparisons.

Another:

It’s not that the researchers performed hundreds of different comparisons and picked ones that were statistically significant. Rather, they start with a somewhat-formed idea in their mind of what comparison to perform, and they refine that idea in light of the data. They saw a pattern in red and pink, and they combined the colors.

Succinctly:

There is a one-to-many mapping from scientific to statistical hypotheses.

And one more, emphasis mine:

In all the cases we have discussed, the published analysis has a story that is consistent with the scientific hypotheses that motivated the work, but other data patterns (which, given the sample sizes, could easily have occurred by chance) would naturally have led to different data analyses (for example, a focus on main effects rather than interactions, or a different choice of data subsets to compare) which equally could have been used to support the research hypotheses. The result remains, as we have written elsewhere, a sort of machine for producing and publicizing random patterns.

In short, it's not that EDA leads to a "spurious hypothesis"; it's that testing a hypothesis with the same dataset that prompted the hypothesis can lead to spurious conclusions. 
If you're interested in conquering this obstacle, Gelman has another paper arguing that many of these problems disappear in a Bayesian framework, and the paper with Loken references "pre-publication replication" as anecdotally described in the first section of this paper.
A: Almost by definition, yes, of course EDA without CDA attracts Texas sharpshooters.
The difficulty when CDA is not possible (perhaps no further data can be obtained) is in being honest with yourself about how many tests you've really performed, and thus in assigning some kind of $p$-value to your discovery.  Even in cases when the search space could in principle be counted, the $p$-value calculation is either done wrongly or not at all: see wikipedia for a notorious example.  
A: Just to add to the already great answers: There is a middle ground between a full CDA and just accepting your EDA results at face value. Once you've found a possible feature of interest (or hypothesis), you can get a sense of its robustness by performing cross-validation (CV) or bootstrap simulations. If your findings depend on only a few key observations, then CV or Bootstrap will show that many of the folds(CV) or boostrap samples fail to reproduce the observed feature.
This is not a foolproof method, but its a good intermediate check before going for a full CDA (or purposefully holding out a "validation set" from your initial data pool).
A: If one views the role of EDA strictly as generating hypotheses, then no the sharpshooter fallacy does not apply. However, it is very important that subsequent confirmatory trials are indeed independent. Many researchers attempt to "reconcile differences" with things like pooled analyses, meta analyses, and Bayesian methods. This means that at least some of the evidence presented in such an analysis includes "the circle around the random bullet holes".
A: This paints a very negative view of exploratory data analysis. While the argument is not wrong, it's really saying "what can go wrong when I use a very important tool in the wrong manner?"
Accepting unadjusted p-values from EDA methods will lead to vastly inflated type I error rates. But I think Tukey would not be happy with anyone doing this. The point of EDA is not to make definitive conclusions about relations in the data, but rather to look for potential novel relations in the data to follow up on. 
Leaving out this step in the larger scientific process is essentially hamstringing science to never be able to find new interesting aspects of our data, outside of pure logical deduction. Ever try to logically deduce how over expression of a set of genes will affect survival of a cell? Hint: it's not very easy (one of our favorite jokes among the bioinformatics staff at my work was when a physicist asked "Why don't you just simulate the physical properties of different gene interactions? It's a finite parameters space.")
Personally, I think confusion about this can lead to a great slow down in scientific progress. I know too many non-statistical researchers that will state that they do not want to do EDA procedures on preliminary data, because they "know that EDA can be bad". 
In conclusion, it's absolutely true that using EDA methods and treating them as confirmatory data analysis methods will lead to invalid results. However, the lack of proper use of EDA can lead to almost no results. 
A: The most rigorous criterion for data model selection is the degree to which is approximates the Kolmogorov Complexity of the data -- which is to say the degree to which it losslessly compress the data.  This can, in theory, result from exploratory data analysis alone.
See "Causal deconvolution by algorithmic generative models"
