0
$\begingroup$

It's maybe some kind of philosophical question. I have a large dataset with information about customers. I return to the dataset from time to time to make some estimations or check different hypothesis.

The longer I look into the data the more probable I will find some spurious insights there. However if I can't know in advance all ideas that will come in my mind and it's hard to track all analysis I and others do with this data. So how can I address type I errors? The paper "Towards sustainable insights" warns that if I haven't checked hypothesis with formal test but made some decision eyeballing the density plots I've already made multiple comparisons. Life is pain.

But should I worry? The Iris dataset is analysed by thousands of people for decades. I believe they already tested everything possible about it. Although it doesn't imply that I mush divide my alpha-level by millions when comparing petal lengths.

$\endgroup$
0
$\begingroup$

For your Iris example, the analyses of many other people is likely not influencing your analysis, whereas this could be happening with your colleagues analyses of your dataset. It is good to have some caution, but it shouldn't paralyze you. There are lots of potential suggestions, but I will offer three:

  1. Consider blind analysis, in which you generate panels of possible results, and consider what questions analyzing each panel triggers. (This addresses your concern "if I can't know in advance all ideas that will come in my mind.") See https://www.nature.com/news/blind-analysis-hide-results-to-seek-the-truth-1.18510

  2. Once you've begun looking at actual results, clearly distinguish analyses planned a priori and post hoc analyses. As a result, you may consider effect sizes more important than p-values in your post-hoc analyses.

  3. Collect new data to test hypotheses developed in post hoc analyses. This is a great solution to some of your worries, when it is possible.

$\endgroup$
  • $\begingroup$ Thank you. I was not aware about the "blind analysis" technique. We will consider it. $\endgroup$ – testlnord Oct 18 '18 at 9:15

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.