Simulation of type-1 error rate, and how to check type-2-error (rate)

Question

With $H_{0}:\mu = 0$,

iter = 100
n = 500
# alpha, beta
a = .05 # b(0.5) = .95 (given in the question)

and $x{\sim}\mathcal{N}(0.5, \sigma)$, for arbitrary $\sigma{>}0$, we apply a statistical test to test $H_{0}$, and expect that

1. 5 tests make a type-1 error, or
2. 5 tests make a type-2 error, or
3. 5 tests reject $H_{0}$?

These options are true/false statements. To check (1), I did

set.seed(1)
replicate(iter, t.test(rnorm(500, .5, 5))$p.value < a) |> sum()

## e.g. returns
# [1] 63

I noticed that the results are heavily dependent on the choice of $\sigma$, i.e. it is difficult to conclude something general for me. I think that neither (1.) nor (3.) are correct. How can one check (2.)?

Answer 1

Your t-test is calculating a p-value for a sample from a population where the mean is 0.5 and the standard deviation is 5. You are testing the null hypothesis that the mean in the population is zero.

You cannot make a type I error. A type I error is the probability of obtaining a result when the null hypothesis is true. The null hypothesis is not true.

If you find a significant result, you (correctly) reject the null hypothesis. If you do not, you fail to reject the null hypothesis and make a type II error. Sorry, but I don't understand 2 and 3.

However, you seem to be doing a simulation to estimate power.

You can do this with the power.t.test() function and get an exact result with probaiblity $a$.

Make the mean 0 (i.e. rnorm(500, 0, 5) and you should obtain a statistically significant result

power.t.test(type = "one.sample", delta = 0.5, n = 500, sd = 5)

Gives me:

     One-sample t test power calculation 

              n = 500
          delta = 0.5
             sd = 5
      sig.level = 0.05
          power = 0.6071117
    alternative = two.sided

Compare with:

iter = 10000
n = 500
# alpha, beta
a = .05 # b(0.5) = .95 (given in the question)
set.seed(1)
replicate(iter, t.test(rnorm(500, .5, 5))$p.value < a) |> sum()

Which gives me 6051 (or 0.6051, pretty close to 0.6071 that I get from power.t.test()).

Sometimes there is no function to estimate power, and then you need to (as you have) do a simulation.