Multiple comparison and secondary research

Question

It's maybe some kind of philosophical question. I have a large dataset with information about customers. I return to the dataset from time to time to make some estimations or check different hypothesis.

The longer I look into the data the more probable I will find some spurious insights there. However if I can't know in advance all ideas that will come in my mind and it's hard to track all analysis I and others do with this data. So how can I address type I errors? The paper "Towards sustainable insights" warns that if I haven't checked hypothesis with formal test but made some decision eyeballing the density plots I've already made multiple comparisons. Life is pain.

But should I worry? The Iris dataset is analysed by thousands of people for decades. I believe they already tested everything possible about it. Although it doesn't imply that I mush divide my alpha-level by millions when comparing petal lengths.

Answer 1

For your Iris example, the analyses of many other people is likely not influencing your analysis, whereas this could be happening with your colleagues analyses of your dataset. It is good to have some caution, but it shouldn't paralyze you. There are lots of potential suggestions, but I will offer three:

1. Consider blind analysis, in which you generate panels of possible results, and consider what questions analyzing each panel triggers. (This addresses your concern "if I can't know in advance all ideas that will come in my mind.") See https://www.nature.com/news/blind-analysis-hide-results-to-seek-the-truth-1.18510

2. Once you've begun looking at actual results, clearly distinguish analyses planned a priori and post hoc analyses. As a result, you may consider effect sizes more important than p-values in your post-hoc analyses.

3. Collect new data to test hypotheses developed in post hoc analyses. This is a great solution to some of your worries, when it is possible.