I was reading this article in Nature in which some fallacies are explained in the context of data analysis. I noticed that the Texas sharpshooter fallacy was particularly difficult to avoid:
A cognitive trap that awaits during data analysis is illustrated by the fable of the Texas sharpshooter: an inept marksman who fires a random pattern of bullets at the side of a barn, draws a target around the biggest clump of bullet holes, and points proudly at his success.
His bullseye is obviously laughable — but the fallacy is not so obvious to gamblers who believe in a 'hot hand' when they have a streak of wins, or to people who see supernatural significance when a lottery draw comes up as all odd numbers.
Nor is it always obvious to researchers. “You just get some encouragement from the data and then think, well, this is the path to go down,” says Pashler. “You don't realize you had 27 different options and you picked the one that gave you the most agreeable or interesting results, and now you're engaged in something that's not at all an unbiased representation of the data.”
I think that kind of exploration work is commonplace and often, hypotheses are constructed based on that part of the analysis. There is a whole approach (EDA) dedicated to this process:
Exploratory data analysis was promoted by John Tukey to encourage statisticians to explore the data, and possibly formulate hypotheses that could lead to new data collection and experiments
It looks like any exploratory process performed without having a hypothesis beforehand is prone to generate spurious hypotheses.
Notice that the description of EDA above actually talks about new data collection and experiments
. I understand that after new data have been collected, then a confirmatory data analysis (CDA) is appropriate. However, I don't think this distinction is made very clearly, and although a separation of EDA and CDA would be ideal, surely there are some circumstances in which this is not feasible. I would go as far as to say that following this separation strictly is uncommon and most practitioners don't subscribe to the EDA paradigm at all.
So my question is: Does EDA (or any informal process of exploring data) make it more likely to fall for the Texas sharpshooter fallacy?