The significance level = alpha level = probability for a type I error is commonly set to 5% when conducting hypothesis testing. This means that if one and the same experiment under the same conditions was repeated a lot of times, it would be expected that a type I error would occur approximately 5% of the times.

But is the intuition correct that this idea could directly be extended to a big number of very different experiments (but all with a hypothesis testing using 5% significance level)? Can I say that of all the statements in scientific papers out there (based on hypothesis testing with alpha = 5%), on average every 20th is incorrect? Is it correct that, as this is a look on the long run or a lot of cases, actually observed p-values in all those papers should have nothing to do with the 1/20 fraction?

  • 8
    $\begingroup$ Your introductory paragraph is mistaken (and therefore so are all the consequences you draw from it), because it implicitly assumes that all published studies concern situations where the null hypothesis is true, which is far from being the case. $\endgroup$ – whuber Mar 15 '17 at 13:07
  • 5
    $\begingroup$ Just adding to @whuber 's comment. IF you did 100 tests of totally nonsensical hypotheses (e.g. is Social Security Number related to street address number? ) then you would get 5% significant results. But that's not what people do. $\endgroup$ – Peter Flom Mar 15 '17 at 13:49
  • $\begingroup$ Green jelly beans linked to acne $\endgroup$ – dotancohen Mar 15 '17 at 17:05
  • $\begingroup$ Published scientific statements are biased towards those claiming significance, pushing up the Type I error rate $\endgroup$ – Henry Mar 15 '17 at 18:37

A type I error of 5% means that you will wrongly reject the null hypothesis 5% of the time, if the null hypothesis is true. Additionally, there is power, the probability that the null hypothesis is correctly rejected, assuming an alternative of a certain strength. This is another type of error that may occur much more often than a type I error.

Additionally, the expected proportion of rejected null hypotheses that is correctly rejected, if rejection is done at the 5% significance level is not 5%. The expected proportion depends on the proportion of the test null hypotheses that is actually true. If 100% of null hypotheses are true, then 100% of rejected null hypotheses are wrongly rejected. If you investigate a lot of true null hypotheses, you will easily still get 30 to 50% or so.

  • 1
    $\begingroup$ Look up the term "False Discovery Rate" to learn more about this. $\endgroup$ – Harvey Motulsky Mar 15 '17 at 15:27

Whuber's comment covered the main issue: type I error requires that the null hypothesis be true, and that condition cannot be assessed by examining p-values. Even without knowing the proportion of studies with null hypotheses that happen to be true, it seems unlikely that it is 100% of them, or that new studies are reliably produced with that same trait.

Even if the above were disregarded, it would be difficult to make comparisons across studies which study different phenomena using different methods. Different studies may well draw from different populations, apply different data transformations, use different null hypotheses, look for different outcomes, and have any number of other differences.

The alpha describes patterns in sample statistics from repeated samples describing parameters for a given population, as per PeterFlom's comment, and means very little when there are differences beyond the composition of random samples in a fixed population.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.