1
$\begingroup$

Here is what I am confused about:

Say we have conducted multiple hypothesis tests on the same experiment. One test gives insignificant p-value only after correction is applied. If I happen to perform this test alone, it will be significant because correction is not needed in such case, right?

It is seems contradictory to me. We got different result for this very variables pair although nothing changed.

$\endgroup$
0
1
$\begingroup$

One should have decided beforehand which tests to perform, and then stick to these tests (and their correction). If you don't report all of the tests you have performed, and instead only report the one test that was significant (without correction), this is a deception known as $p$-value fishing (part of the bigger problem of $p$-value hacking).

Because of the definition of the $p$-value and significance, there's always a chance to find something "significant" by chance (typically one allows 0.05). If you do 100 different hypthesis tests on the same data, there's a good chance that a few of them are "significant" just by chance. Multiple testing correction is a solution to this problem.

A good read is this blog post by Andrew Gelman.

$\endgroup$
1
  • 2
    $\begingroup$ It gets worse: If you conduct 100 experiments to explore a completely bogus hypothesis, and then on each dataset you perform only one test, you can still expect five of these to be significant. Then you get exactly these five published, because of the "exciting results", and the rest of the experiments are never to be heard of again. Unfortunately, that is very close to reality. Most scientists work for years on one topic, examining a limited set of (probably interrelated) hypotheses, and journals still love significant results and the rest gets thrown away. This is called publication bias. $\endgroup$ – morphist Apr 18 '19 at 15:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.