Is this interpretation of the relationship between p-values and type I error rates correct? I am trying to check my understanding of the relationship between p-values and type I error rates. I am going to use the following example to do so, so I would appreciate any feedback on it.
Suppose we are given an i.i.d. sample $\mathbf{X}$ of size $n$ from $N(\mu,\sigma^2)$, where $\mu = 0$ and $\sigma^2 = 1$. Suppose that we did not know the true values of $\mu$ and $\sigma^2$ beforehand, and that we want to test the null hypothesis
\begin{align}
H_0 &: \mu = 0
\end{align}
This can be done using a simple $t$-test. If $\overline{\mathbf{X}}$ is the sample mean of $\mathbf{X}$ and $\hat{\sigma}^2$ is its sample variance, then we define the test statistic $T(\mathbf{X},\mu)$ as
$$
T(\mathbf{X},\mu) = \frac{\overline{\mathbf{X}} - \mu}{\sqrt{\hat{\sigma}^2 / n}}
$$
Under the null hypothesis, this test statistic becomes
$$
T(\mathbf{X},0) = \frac{\overline{\mathbf{X}}}{\sqrt{\hat{\sigma}^2 / n}}
$$
which is $t$-distributed with $n$ degrees of freedom. To decide whether to reject $H_0$ or not, we compute a p-value, and if it is less than some significance level $\alpha$ (say 0.05), then we reject $H_0$. However, as p-values are random variables that depend on the sample $\mathbf{X}$, then there is a chance that we observe some sample $\mathbf{X}$ for which we compute a p-value that is less than $\alpha$, and therefore reject $H_0$, even though it is actually true. This chance is the type I error rate.
Is my understanding, as demonstrated by this example, correct?
 A: Yes, you're on the right track.
If you have a random sample z of size $n=40$ from a normal distribution and you want
to test whether it comes from a normal distribution with $\mu=0,$ then
you are not likely to reject $H_0: \mu = 0$ at the 5% level; your P-value will likely be larger
than $0.05,$ as below (using R):
set.seed(123)
z = rnorm(40)
t.test(z, mu=0)

        One Sample t-test

data:  z
t = 0.3183, df = 39, p-value = 0.752
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
 -0.2419421  0.3323088
sample estimates:
 mean of x 
0.04518332 

Moreover, if you repeat such a test 100,000 times, then
you will reject $H_0,$ with an unusually small P-value (below $0.05)$
only about 5% of the time:
set.seed(2022)
pv = replicate(10^5, t.test(rnorm(40), mu=0)$p.val)
mean(pv <= 0.05)
[1] 0.05028

For such tests, the P-value has a uniform distribution on $(0,1):$
hist(pv, prob=T, col="skyblue2")


However, if you sample from a shifted exponential distribution with mean $0$ and variance $1,$ then the t test does not give accurate
P-values, and the true "significance level" of resulting t tests will not be 5% (closer to 7% shown below).
set.seed(506)
pv = replicate(10^5, t.test(rexp(40)-1, mu=0)$p.val)
mean(pv <= 0.05)
[1] 0.06817

In particular, P-values from these inappropriate t tests will not
be uniformly distributed under $H_0: \mu = 0.$
hist(pv, prob=T, col="skyblue2")


Note: Imprecise discussions in some elementary statistics texts have
lead some students to believe that t tests "always"
apply, if $n \ge 30$--whether or not data are normal. But as we see here, that claim is not true.
