# Power of a test with relation to p-value, linear regression

Let's assume a linear regression $$y_i = \beta_0 + \beta_1 x_{i,1} + \beta_2 x_{i,2} + \varepsilon_i$$ where $$\varepsilon_i, x_1, x_2 \sim N(0,1)$$ and $$\beta_0 = \beta_2 = 0.5, \beta_1 = 1$$. Given the null hypothesis \begin{align} H_0: \beta_i = 0 \\ H_1: \beta_i \neq 0 \end{align} and simultaneously the t statistics $$t = \dfrac{\hat \beta_i}{se(\hat \beta_i)}$$ How to assess the power of the t-test? In this the alternative hypothesis is simply a negation, so I don't know the distribution under $$H_1$$. I could generate, let's say, $$N$$ datasets with $$n$$ observations each, but what next? I thought about setting something like this: for each $$i$$ in $$1$$ to $$N$$: $$Power(i) = 1$$ if and only if $$p-value$$ is lower than $$\alpha$$ and $$0$$ otherwise. Then, $$Power = \dfrac{\sum_i Power(i)}{N}$$ However, I am not convinced to that, I can't really convince myself that this could be the right way.

• How would you calculate the power in a really simple setting, such as a one-sample t-test?
– Dave
Commented Oct 16, 2023 at 15:57

What you describe is basically a Monte Carlo study for simulating power. In some cases one can also analytically calculate exact power, see e.g. Power calculations Uniform(0,$\theta$) or "Unbiased" hypothesis test --- what does it mean actually?.

Based on the asymptotic distribution of the estimator it is sometimes also possible to compute power against so-called "local alternatives". These make the testing problem "harder" by moving the alternative closer to the null at a suitable rate. This device ensures that the answer, asymptotically, isn't always simply that the power of a somewhat decent test is 1 against some fixed alternative (a fairly weak concept known as test consistency)

I would prefer writing the terms in your sum as "rejection" rather than $$Power(i)$$, because power is probability that you estimate with the proportion of rejections (0/1 events) over the Monte Carlo runs $$i=1,\ldots,N$$.

And indeed, you then need to specify (as your numerical values do) how exactly the parameters look like under the alternative. The power will depend on these choices. For example, if you test the first regressor (replace coef[3,4] with coef[2,4]), power will be higher as that value is further away from the null value.

Here is a code snippet for testing the second coefficient:

mc.func <- function(n){
x1 <- rnorm(n)
x2 <- rnorm(n)
eps <- rnorm(n)
y <- 0.5 + 1*x1 + 0.5*x2 + eps
summary(lm(y~x1+x2))\$coef[3,4] < 0.05
}

power <- mean(replicate(5000, mc.func(50)))