# Frequentist properties of p-values in relation to type I error

This question is related to frequentist properties of p-values and their relation to type I error and why the results from an online simulation differ from what I would have expected.

Assume that I perform an experiment and do hypothesis testing at a significance level of 0.05. Next, I compute the p-value. If it is less than 0.05 then I reject the null hypothesis, if it is greater than 0.05, then I accept the null hypothesis (as per Neyman-Pearson hypothesis testing). Now, if I repeated this experiment hundreds of times (each time either accepting or rejecting the null hypothesis at 0.05), then the type I error (chance of rejecting a true null hypothesis) should be around 5% is that not correct?

I wanted to test my understanding so I used this java applet: http://www.stat.duke.edu/~berger/applet2/pvalue.html to simulate exactly such an experiment. I kept everything at their default levels in the applet except in the top bar where I changed the range of p-values from 0 to 0.05. Essentially, this is allowing me to 'reject' all those experiments where the p-value was < 0.05 and find out how many H0 were incorrectly rejected (H0 was actually true) and how many H0 were correctly rejected (H1 was actually true).

I would have assumed that I would get around 5% true nulls; however, when I ran it, I get around 12% H0, and 88% H1, which means that 12% of the numbers I rejected were true nulls, while 88% were false, this is a type 1 error of 12%. What am I missing? Can someone please explain why the applet came up with these results?

• I was not able to run your applet. However, came across this which gives a good understanding of p-value. – danas.zuokas Aug 6 '12 at 7:46
• Did you run it both for $H_0$ true and for $H_1$ true before computing the rejection rate? You should get a $5~\%$ rejection rate if you only run it under $H_0$. That's what is meant by type I error rate. The probability of rejection is larger under $H_1$, so if you run it under both hypotheses the rejection rate should be above $5~\%$ (depending on how many times you run it under each hypothesis). – MånsT Aug 6 '12 at 7:48
• I agree with MansT. It sounds like you have a misunderstanding above the type I error and the p-value. You should be testing under the null hypothesis. Under the null hypothesis the p-value is uniform on [0, 1] and consequently will be below 0.05 5% of the time. Under an alternative hypothesis the distribution of the p-value will not be uniform and will tend to be higher than 0.05. – Michael R. Chernick Aug 6 '12 at 11:14
• @MånsT, hmm, how would I run it under only the null hypothesis? The applet does not give me an option to do that as far as I can tell :S – BYS2 Aug 6 '12 at 15:01
• @MichaelChernick I'm not quite sure I follow, doesn't the p-value depend solely on the distribution assuming the null hypothesis is true? The non-central alternative distribution only comes into play for power and sample size as far as I know – BYS2 Aug 6 '12 at 23:54

I can't for the life of me get that applet to run in my browser, so I'll try to give an example using R instead.

As noted in the comments, it seems that what caused the confusion is that the applet runs under both the alternative and the null hypothesis. To check that the type I error rate really is $0.05$ you need to run it under the null hypothesis only.

Here is an example where we use the $t$-test to test whether the mean $\mu$ of a normal distribution equals $0$. That is, we test $H_0: \mu=0.$ We simulate $10,000$ samples from ${\rm N}(0,\sigma^2)$ and compute the $p$-value for each sample.

We also simulate $10,000$ samples from the ${\rm N}(0.25,\sigma^2)$ and ${\rm N}(0.5,\sigma^2)$ distributions and compute the $p$-values.

set.seed(201208)
B<-10000
p.values1<-p.values2<-p.values3<-vector(length=B)

for(i in 1:B)
{
x1<-rnorm(25)
p.values1[i]<-t.test(x1)$p.value x2<-rnorm(25,0.25) p.values2[i]<-t.test(x2)$p.value

x3<-rnorm(25,0.5)
p.values3[i]<-t.test(x3)$p.value }  We can now compute the proportion of samples that lead to a rejection of$H_0: \mu=0$at the$5~\%$level: sum(p.values1<=0.05)/B sum(p.values2<=0.05)/B sum(p.values3<=0.05)/B  In this case, the answers are$0.505$under the null hypothesis ($\approx 0.05$, just as we would expect!),$0.2187$when$\mu=0.25$and$0.6754$when$\mu=0.5$. We can visualize the results by plotting histograms of the$p$-values: For$\mu=0$, the$p$-values are uniformly distributed on$\lbrack 0,1\rbrack$. Under the alternatives, the distribution of the$p$-values has more mass closer to$0$(more so the further away from$0$that$\mu$is). We can also compare the distribution of the$p$-values using box-and-whiskers-plots: Hopefully it is clear from the picture that the probability of rejection, i.e. the probability that the$p$-value is lower than$0.05$depends on whether the null hypothesis or an alternative hypothesis is true. In this case, you should only expect the rejection rate to be$0.05$when$\mu=0\$.

The code for producing these plots is:

#Boxplots:
boxplot(p.values1,p.values2,p.values3,names=c("mu=0","mu=0.25","mu=0.5"))

# Histograms:
par(mfrow=c(1,3))
hist(p.values1,main="mu=0")
hist(p.values2,main="mu=0.25")
hist(p.values3,main="mu=0.5")

• Brilliant, that was a great explanation! One question though, you say "In this case, the answers are 0.505 under the null hypothesis (≈0.05, just as we would expect!), 0.2187 when μ=0.25..." Don't you mean 0.0505 (and not 0.505) under the null hypothesis? and not sure why you can't access the applet :(, the main page is stat.duke.edu/~berger/p-values.html though in case you want to check it out – BYS2 Aug 8 '12 at 13:09
• @BYS2: You're quite right, it should of course be 0.0505! – MånsT Aug 8 '12 at 13:13