Is this interpretation of the relationship between p-values and type I error rates correct?

Question

I am trying to check my understanding of the relationship between p-values and type I error rates. I am going to use the following example to do so, so I would appreciate any feedback on it.

Suppose we are given an i.i.d. sample $\mathbf{X}$ of size $n$ from $N(\mu,\sigma^2)$, where $\mu = 0$ and $\sigma^2 = 1$. Suppose that we did not know the true values of $\mu$ and $\sigma^2$ beforehand, and that we want to test the null hypothesis \begin{align} H_0 &: \mu = 0 \end{align} This can be done using a simple $t$-test. If $\overline{\mathbf{X}}$ is the sample mean of $\mathbf{X}$ and $\hat{\sigma}^2$ is its sample variance, then we define the test statistic $T(\mathbf{X},\mu)$ as $$ T(\mathbf{X},\mu) = \frac{\overline{\mathbf{X}} - \mu}{\sqrt{\hat{\sigma}^2 / n}} $$ Under the null hypothesis, this test statistic becomes $$ T(\mathbf{X},0) = \frac{\overline{\mathbf{X}}}{\sqrt{\hat{\sigma}^2 / n}} $$ which is $t$-distributed with $n$ degrees of freedom. To decide whether to reject $H_0$ or not, we compute a p-value, and if it is less than some significance level $\alpha$ (say 0.05), then we reject $H_0$. However, as p-values are random variables that depend on the sample $\mathbf{X}$, then there is a chance that we observe some sample $\mathbf{X}$ for which we compute a p-value that is less than $\alpha$, and therefore reject $H_0$, even though it is actually true. This chance is the type I error rate.

Is my understanding, as demonstrated by this example, correct?

Answer 1

Yes, you're on the right track.

If you have a random sample z of size $n=40$ from a normal distribution and you want to test whether it comes from a normal distribution with $\mu=0,$ then you are not likely to reject $H_0: \mu = 0$ at the 5% level; your P-value will likely be larger than $0.05,$ as below (using R):

set.seed(123)
z = rnorm(40)
t.test(z, mu=0)

        One Sample t-test

data:  z
t = 0.3183, df = 39, p-value = 0.752
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
 -0.2419421  0.3323088
sample estimates:
 mean of x 
0.04518332 

Moreover, if you repeat such a test 100,000 times, then you will reject $H_0,$ with an unusually small P-value (below $0.05)$ only about 5% of the time:

set.seed(2022)
pv = replicate(10^5, t.test(rnorm(40), mu=0)$p.val)
mean(pv <= 0.05)
[1] 0.05028

For such tests, the P-value has a uniform distribution on $(0,1):$

hist(pv, prob=T, col="skyblue2")

However, if you sample from a shifted exponential distribution with mean $0$ and variance $1,$ then the t test does not give accurate P-values, and the true "significance level" of resulting t tests will not be 5% (closer to 7% shown below).

set.seed(506)
pv = replicate(10^5, t.test(rexp(40)-1, mu=0)$p.val)
mean(pv <= 0.05)
[1] 0.06817

In particular, P-values from these inappropriate t tests will not be uniformly distributed under $H_0: \mu = 0.$

hist(pv, prob=T, col="skyblue2")

Note: Imprecise discussions in some elementary statistics texts have lead some students to believe that t tests "always" apply, if $n \ge 30$--whether or not data are normal. But as we see here, that claim is not true.