# Is this interpretation of the relationship between p-values and type I error rates correct?

I am trying to check my understanding of the relationship between p-values and type I error rates. I am going to use the following example to do so, so I would appreciate any feedback on it.

Suppose we are given an i.i.d. sample $$\mathbf{X}$$ of size $$n$$ from $$N(\mu,\sigma^2)$$, where $$\mu = 0$$ and $$\sigma^2 = 1$$. Suppose that we did not know the true values of $$\mu$$ and $$\sigma^2$$ beforehand, and that we want to test the null hypothesis \begin{align} H_0 &: \mu = 0 \end{align} This can be done using a simple $$t$$-test. If $$\overline{\mathbf{X}}$$ is the sample mean of $$\mathbf{X}$$ and $$\hat{\sigma}^2$$ is its sample variance, then we define the test statistic $$T(\mathbf{X},\mu)$$ as $$T(\mathbf{X},\mu) = \frac{\overline{\mathbf{X}} - \mu}{\sqrt{\hat{\sigma}^2 / n}}$$ Under the null hypothesis, this test statistic becomes $$T(\mathbf{X},0) = \frac{\overline{\mathbf{X}}}{\sqrt{\hat{\sigma}^2 / n}}$$ which is $$t$$-distributed with $$n$$ degrees of freedom. To decide whether to reject $$H_0$$ or not, we compute a p-value, and if it is less than some significance level $$\alpha$$ (say 0.05), then we reject $$H_0$$. However, as p-values are random variables that depend on the sample $$\mathbf{X}$$, then there is a chance that we observe some sample $$\mathbf{X}$$ for which we compute a p-value that is less than $$\alpha$$, and therefore reject $$H_0$$, even though it is actually true. This chance is the type I error rate.

Is my understanding, as demonstrated by this example, correct?

Yes, you're on the right track.

If you have a random sample z of size $$n=40$$ from a normal distribution and you want to test whether it comes from a normal distribution with $$\mu=0,$$ then you are not likely to reject $$H_0: \mu = 0$$ at the 5% level; your P-value will likely be larger than $$0.05,$$ as below (using R):

set.seed(123)
z = rnorm(40)
t.test(z, mu=0)

One Sample t-test

data:  z
t = 0.3183, df = 39, p-value = 0.752
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
-0.2419421  0.3323088
sample estimates:
mean of x
0.04518332


Moreover, if you repeat such a test 100,000 times, then you will reject $$H_0,$$ with an unusually small P-value (below $$0.05)$$ only about 5% of the time:

set.seed(2022)
pv = replicate(10^5, t.test(rnorm(40), mu=0)$p.val) mean(pv <= 0.05) [1] 0.05028  For such tests, the P-value has a uniform distribution on $$(0,1):$$ hist(pv, prob=T, col="skyblue2")  However, if you sample from a shifted exponential distribution with mean $$0$$ and variance $$1,$$ then the t test does not give accurate P-values, and the true "significance level" of resulting t tests will not be 5% (closer to 7% shown below). set.seed(506) pv = replicate(10^5, t.test(rexp(40)-1, mu=0)$p.val)
mean(pv <= 0.05)
[1] 0.06817


In particular, P-values from these inappropriate t tests will not be uniformly distributed under $$H_0: \mu = 0.$$

hist(pv, prob=T, col="skyblue2")


Note: Imprecise discussions in some elementary statistics texts have lead some students to believe that t tests "always" apply, if $$n \ge 30$$--whether or not data are normal. But as we see here, that claim is not true.