I am trying to check my understanding of the relationship between p-values and type I error rates. I am going to use the following example to do so, so I would appreciate any feedback on it.

Suppose we are given an i.i.d. sample $\mathbf{X}$ of size $n$ from $N(\mu,\sigma^2)$, where $\mu = 0$ and $\sigma^2 = 1$. Suppose that we did not know the true values of $\mu$ and $\sigma^2$ beforehand, and that we want to test the null hypothesis \begin{align} H_0 &: \mu = 0 \end{align} This can be done using a simple $t$-test. If $\overline{\mathbf{X}}$ is the sample mean of $\mathbf{X}$ and $\hat{\sigma}^2$ is its sample variance, then we define the test statistic $T(\mathbf{X},\mu)$ as $$ T(\mathbf{X},\mu) = \frac{\overline{\mathbf{X}} - \mu}{\sqrt{\hat{\sigma}^2 / n}} $$ Under the null hypothesis, this test statistic becomes $$ T(\mathbf{X},0) = \frac{\overline{\mathbf{X}}}{\sqrt{\hat{\sigma}^2 / n}} $$ which is $t$-distributed with $n$ degrees of freedom. To decide whether to reject $H_0$ or not, we compute a p-value, and if it is less than some significance level $\alpha$ (say 0.05), then we reject $H_0$. However, as p-values are random variables that depend on the sample $\mathbf{X}$, then there is a chance that we observe some sample $\mathbf{X}$ for which we compute a p-value that is less than $\alpha$, and therefore reject $H_0$, even though it is actually true. This chance is the type I error rate.

Is my understanding, as demonstrated by this example, correct?


1 Answer 1


Yes, you're on the right track.

If you have a random sample z of size $n=40$ from a normal distribution and you want to test whether it comes from a normal distribution with $\mu=0,$ then you are not likely to reject $H_0: \mu = 0$ at the 5% level; your P-value will likely be larger than $0.05,$ as below (using R):

z = rnorm(40)
t.test(z, mu=0)

        One Sample t-test

data:  z
t = 0.3183, df = 39, p-value = 0.752
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
 -0.2419421  0.3323088
sample estimates:
 mean of x 

Moreover, if you repeat such a test 100,000 times, then you will reject $H_0,$ with an unusually small P-value (below $0.05)$ only about 5% of the time:

pv = replicate(10^5, t.test(rnorm(40), mu=0)$p.val)
mean(pv <= 0.05)
[1] 0.05028

For such tests, the P-value has a uniform distribution on $(0,1):$

hist(pv, prob=T, col="skyblue2")

enter image description here

However, if you sample from a shifted exponential distribution with mean $0$ and variance $1,$ then the t test does not give accurate P-values, and the true "significance level" of resulting t tests will not be 5% (closer to 7% shown below).

pv = replicate(10^5, t.test(rexp(40)-1, mu=0)$p.val)
mean(pv <= 0.05)
[1] 0.06817

In particular, P-values from these inappropriate t tests will not be uniformly distributed under $H_0: \mu = 0.$

hist(pv, prob=T, col="skyblue2")

enter image description here

Note: Imprecise discussions in some elementary statistics texts have lead some students to believe that t tests "always" apply, if $n \ge 30$--whether or not data are normal. But as we see here, that claim is not true.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.