5
$\begingroup$

This question is related to frequentist properties of p-values and their relation to type I error and why the results from an online simulation differ from what I would have expected.

Assume that I perform an experiment and do hypothesis testing at a significance level of 0.05. Next, I compute the p-value. If it is less than 0.05 then I reject the null hypothesis, if it is greater than 0.05, then I accept the null hypothesis (as per Neyman-Pearson hypothesis testing). Now, if I repeated this experiment hundreds of times (each time either accepting or rejecting the null hypothesis at 0.05), then the type I error (chance of rejecting a true null hypothesis) should be around 5% is that not correct?

I wanted to test my understanding so I used this java applet: http://www.stat.duke.edu/~berger/applet2/pvalue.html to simulate exactly such an experiment. I kept everything at their default levels in the applet except in the top bar where I changed the range of p-values from 0 to 0.05. Essentially, this is allowing me to 'reject' all those experiments where the p-value was < 0.05 and find out how many H0 were incorrectly rejected (H0 was actually true) and how many H0 were correctly rejected (H1 was actually true).

I would have assumed that I would get around 5% true nulls; however, when I ran it, I get around 12% H0, and 88% H1, which means that 12% of the numbers I rejected were true nulls, while 88% were false, this is a type 1 error of 12%. What am I missing? Can someone please explain why the applet came up with these results?

$\endgroup$
  • $\begingroup$ I was not able to run your applet. However, came across this which gives a good understanding of p-value. $\endgroup$ – danas.zuokas Aug 6 '12 at 7:46
  • 2
    $\begingroup$ Did you run it both for $H_0$ true and for $H_1$ true before computing the rejection rate? You should get a $5~\%$ rejection rate if you only run it under $H_0$. That's what is meant by type I error rate. The probability of rejection is larger under $H_1$, so if you run it under both hypotheses the rejection rate should be above $5~\%$ (depending on how many times you run it under each hypothesis). $\endgroup$ – MånsT Aug 6 '12 at 7:48
  • $\begingroup$ I agree with MansT. It sounds like you have a misunderstanding above the type I error and the p-value. You should be testing under the null hypothesis. Under the null hypothesis the p-value is uniform on [0, 1] and consequently will be below 0.05 5% of the time. Under an alternative hypothesis the distribution of the p-value will not be uniform and will tend to be higher than 0.05. $\endgroup$ – Michael Chernick Aug 6 '12 at 11:14
  • $\begingroup$ @MånsT, hmm, how would I run it under only the null hypothesis? The applet does not give me an option to do that as far as I can tell :S $\endgroup$ – BYS2 Aug 6 '12 at 15:01
  • $\begingroup$ @MichaelChernick I'm not quite sure I follow, doesn't the p-value depend solely on the distribution assuming the null hypothesis is true? The non-central alternative distribution only comes into play for power and sample size as far as I know $\endgroup$ – BYS2 Aug 6 '12 at 23:54
7
$\begingroup$

I can't for the life of me get that applet to run in my browser, so I'll try to give an example using R instead.

As noted in the comments, it seems that what caused the confusion is that the applet runs under both the alternative and the null hypothesis. To check that the type I error rate really is $0.05$ you need to run it under the null hypothesis only.

Here is an example where we use the $t$-test to test whether the mean $\mu$ of a normal distribution equals $0$. That is, we test $H_0: \mu=0.$ We simulate $10,000$ samples from ${\rm N}(0,\sigma^2)$ and compute the $p$-value for each sample.

We also simulate $10,000$ samples from the ${\rm N}(0.25,\sigma^2)$ and ${\rm N}(0.5,\sigma^2)$ distributions and compute the $p$-values.

set.seed(201208)
B<-10000
p.values1<-p.values2<-p.values3<-vector(length=B)

for(i in 1:B)
{
    x1<-rnorm(25)
    p.values1[i]<-t.test(x1)$p.value

    x2<-rnorm(25,0.25)
    p.values2[i]<-t.test(x2)$p.value

    x3<-rnorm(25,0.5)
    p.values3[i]<-t.test(x3)$p.value
}   

We can now compute the proportion of samples that lead to a rejection of $H_0: \mu=0$ at the $5~\%$ level:

sum(p.values1<=0.05)/B
sum(p.values2<=0.05)/B
sum(p.values3<=0.05)/B

In this case, the answers are $0.505$ under the null hypothesis ($\approx 0.05$, just as we would expect!), $0.2187$ when $\mu=0.25$ and $0.6754$ when $\mu=0.5$.

We can visualize the results by plotting histograms of the $p$-values: Histograms

For $\mu=0$, the $p$-values are uniformly distributed on $\lbrack 0,1\rbrack$. Under the alternatives, the distribution of the $p$-values has more mass closer to $0$ (more so the further away from $0$ that $\mu$ is).

We can also compare the distribution of the $p$-values using box-and-whiskers-plots: Boxplots

Hopefully it is clear from the picture that the probability of rejection, i.e. the probability that the $p$-value is lower than $0.05$ depends on whether the null hypothesis or an alternative hypothesis is true. In this case, you should only expect the rejection rate to be $0.05$ when $\mu=0$.

The code for producing these plots is:

#Boxplots:
boxplot(p.values1,p.values2,p.values3,names=c("mu=0","mu=0.25","mu=0.5"))

# Histograms:
par(mfrow=c(1,3))
hist(p.values1,main="mu=0")
hist(p.values2,main="mu=0.25")
hist(p.values3,main="mu=0.5")
$\endgroup$
  • $\begingroup$ Brilliant, that was a great explanation! One question though, you say "In this case, the answers are 0.505 under the null hypothesis (≈0.05, just as we would expect!), 0.2187 when μ=0.25..." Don't you mean 0.0505 (and not 0.505) under the null hypothesis? and not sure why you can't access the applet :(, the main page is stat.duke.edu/~berger/p-values.html though in case you want to check it out $\endgroup$ – BYS2 Aug 8 '12 at 13:09
  • $\begingroup$ @BYS2: You're quite right, it should of course be 0.0505! $\endgroup$ – MånsT Aug 8 '12 at 13:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.