Are on average 5 of 100 scientific statements that base on a 5% significance level incorrect? The significance level = alpha level = probability for a type I error is commonly set to 5% when conducting hypothesis testing. This means that if one and the same experiment under the same conditions was repeated a lot of times, it would be expected that a type I error would occur approximately 5% of the times.
But is the intuition correct that this idea could directly be extended to a big number of very different experiments (but all with a hypothesis testing using 5% significance level)? Can I say that of all the statements in scientific papers out there (based on hypothesis testing with alpha = 5%), on average every 20th is incorrect? Is it correct that, as this is a look on the long run or a lot of cases, actually observed p-values in all those papers should have nothing to do with the 1/20 fraction?
 A: A type I error of 5% means that you will wrongly reject the null hypothesis 5% of the time, if the null hypothesis is true. Additionally, there is power, the probability that the null hypothesis is correctly rejected, assuming an alternative of a certain strength. This is another type of error that may occur much more often than a type I error.
Additionally, the expected proportion of rejected null hypotheses that is correctly rejected, if rejection is done at the 5% significance level is not 5%. The expected proportion depends on the proportion of the test null hypotheses that is actually true. If 100% of null hypotheses are true, then 100% of rejected null hypotheses are wrongly rejected. If you investigate a lot of true null hypotheses, you will easily still get 30 to 50% or so. 
A: Whuber's comment covered the main issue: type I error requires that the null hypothesis be true, and that condition cannot be assessed by examining p-values. Even without knowing the proportion of studies with null hypotheses that happen to be true, it seems unlikely that it is 100% of them, or that new studies are reliably produced with that same trait.
Even if the above were disregarded, it would be difficult to make comparisons across studies which study different phenomena using different methods. Different studies may well draw from different populations, apply different data transformations, use different null hypotheses, look for different outcomes, and have any number of other differences.
The alpha describes patterns in sample statistics from repeated samples describing parameters for a given population, as per PeterFlom's comment, and means very little when there are differences beyond the composition of random samples in a fixed population.
