6
$\begingroup$

It's often claimed that a good practice in statistics is to make adjustments for simultaneous inference to control so-called family-wise error rates. So for example if we did 100 tests some would say we should use more stringent $p$-values to avoid making at least one error, perhaps by using a Bonferroni correction or something similar.

But if we performed those tests on different days, no one would care to make such an adjustment. Why when we perform many tests at once is it important to control family-wise error rates but not if we spread them out? In the latter case do we not simply accept that mistakes are going to be made, so why not in the former?

$\endgroup$
3
  • 1
    $\begingroup$ Related: stats.stackexchange.com/a/16784. $\endgroup$ – amoeba Mar 9 '16 at 14:47
  • $\begingroup$ Even more related: stats.stackexchange.com/questions/1458. Actually, the top answer by John is more or less answering exactly your question, so I am wondering if your Q should not perhaps be closed as a duplicate of that one. $\endgroup$ – amoeba Mar 9 '16 at 22:45
  • $\begingroup$ Yeah, I think it's safe to call it a duplicate. Thanks for the link. $\endgroup$ – dsaxton Mar 9 '16 at 23:55