The significance level = alpha level = probability for a type I error is commonly set to 5% when conducting hypothesis testing. This means that if one and the same experiment under the same conditions was repeated a lot of times, it would be expected that a type I error would occur approximately 5% of the times.
But is the intuition correct that this idea could directly be extended to a big number of very different experiments (but all with a hypothesis testing using 5% significance level)? Can I say that of all the statements in scientific papers out there (based on hypothesis testing with alpha = 5%), on average every 20th is incorrect? Is it correct that, as this is a look on the long run or a lot of cases, actually observed p-values in all those papers should have nothing to do with the 1/20 fraction?