Frequentist properties of p-values in relation to type I error

Question

This question is related to frequentist properties of p-values and their relation to type I error and why the results from an online simulation differ from what I would have expected.

Assume that I perform an experiment and do hypothesis testing at a significance level of 0.05. Next, I compute the p-value. If it is less than 0.05 then I reject the null hypothesis, if it is greater than 0.05, then I accept the null hypothesis (as per Neyman-Pearson hypothesis testing). Now, if I repeated this experiment hundreds of times (each time either accepting or rejecting the null hypothesis at 0.05), then the type I error (chance of rejecting a true null hypothesis) should be around 5% is that not correct?

I wanted to test my understanding so I used this java applet: http://www.stat.duke.edu/~berger/applet2/pvalue.html to simulate exactly such an experiment. I kept everything at their default levels in the applet except in the top bar where I changed the range of p-values from 0 to 0.05. Essentially, this is allowing me to 'reject' all those experiments where the p-value was < 0.05 and find out how many H0 were incorrectly rejected (H0 was actually true) and how many H0 were correctly rejected (H1 was actually true).

I would have assumed that I would get around 5% true nulls; however, when I ran it, I get around 12% H0, and 88% H1, which means that 12% of the numbers I rejected were true nulls, while 88% were false, this is a type 1 error of 12%. What am I missing? Can someone please explain why the applet came up with these results?

Answer 1

I can't for the life of me get that applet to run in my browser, so I'll try to give an example using R instead.

As noted in the comments, it seems that what caused the confusion is that the applet runs under both the alternative and the null hypothesis. To check that the type I error rate really is $0.05$ you need to run it under the null hypothesis only.

Here is an example where we use the $t$-test to test whether the mean $\mu$ of a normal distribution equals $0$. That is, we test $H_0: \mu=0.$ We simulate $10,000$ samples from ${\rm N}(0,\sigma^2)$ and compute the $p$-value for each sample.

We also simulate $10,000$ samples from the ${\rm N}(0.25,\sigma^2)$ and ${\rm N}(0.5,\sigma^2)$ distributions and compute the $p$-values.

set.seed(201208)
B<-10000
p.values1<-p.values2<-p.values3<-vector(length=B)

for(i in 1:B)
{
    x1<-rnorm(25)
    p.values1[i]<-t.test(x1)$p.value

    x2<-rnorm(25,0.25)
    p.values2[i]<-t.test(x2)$p.value

    x3<-rnorm(25,0.5)
    p.values3[i]<-t.test(x3)$p.value
}   

We can now compute the proportion of samples that lead to a rejection of $H_0: \mu=0$ at the $5~\%$ level:

sum(p.values1<=0.05)/B
sum(p.values2<=0.05)/B
sum(p.values3<=0.05)/B

In this case, the answers are $0.505$ under the null hypothesis ($\approx 0.05$, just as we would expect!), $0.2187$ when $\mu=0.25$ and $0.6754$ when $\mu=0.5$.

We can visualize the results by plotting histograms of the $p$-values:

For $\mu=0$, the $p$-values are uniformly distributed on $\lbrack 0,1\rbrack$. Under the alternatives, the distribution of the $p$-values has more mass closer to $0$ (more so the further away from $0$ that $\mu$ is).

We can also compare the distribution of the $p$-values using box-and-whiskers-plots:

Hopefully it is clear from the picture that the probability of rejection, i.e. the probability that the $p$-value is lower than $0.05$ depends on whether the null hypothesis or an alternative hypothesis is true. In this case, you should only expect the rejection rate to be $0.05$ when $\mu=0$.

The code for producing these plots is:

#Boxplots:
boxplot(p.values1,p.values2,p.values3,names=c("mu=0","mu=0.25","mu=0.5"))

# Histograms:
par(mfrow=c(1,3))
hist(p.values1,main="mu=0")
hist(p.values2,main="mu=0.25")
hist(p.values3,main="mu=0.5")