This question is related to frequentist properties of p-values and their relation to type I error and why the results from an online simulation differ from what I would have expected.
Assume that I perform an experiment and do hypothesis testing at a significance level of 0.05. Next, I compute the p-value. If it is less than 0.05 then I reject the null hypothesis, if it is greater than 0.05, then I accept the null hypothesis (as per Neyman-Pearson hypothesis testing). Now, if I repeated this experiment hundreds of times (each time either accepting or rejecting the null hypothesis at 0.05), then the type I error (chance of rejecting a true null hypothesis) should be around 5% is that not correct?
I wanted to test my understanding so I used this java applet: http://www.stat.duke.edu/~berger/applet2/pvalue.html to simulate exactly such an experiment. I kept everything at their default levels in the applet except in the top bar where I changed the range of p-values from 0 to 0.05. Essentially, this is allowing me to 'reject' all those experiments where the p-value was < 0.05 and find out how many H0 were incorrectly rejected (H0 was actually true) and how many H0 were correctly rejected (H1 was actually true).
I would have assumed that I would get around 5% true nulls; however, when I ran it, I get around 12% H0, and 88% H1, which means that 12% of the numbers I rejected were true nulls, while 88% were false, this is a type 1 error of 12%. What am I missing? Can someone please explain why the applet came up with these results?