How statistical packages calculate power

Question

How do statistical packages calculate power? For example, suppose we have a sample $X$ of $100$ observations. We assume that they are from a normally distributed population (iid). Our hypothesis test is $H_{0}: \mu = 6$ vs. $H_a: \mu \neq 6$. Suppose the null hypothesis is rejected. How does a software package compute the probability of correctly rejecting the null hypothesis? Wouldn't it have to repeatedly perform the same hypothesis test on different samples? Or is it possible to compute the power by only performing the hypothesis test on one sample?

Answer 1

You're correct that one way to perform a power analysis would be via simulation. You could repeatedly generate 100 observations from a distribution and test those observations against the null hypothesis. If it consistently reported a (true) difference, you could say the test has high power.

However, for many common tests, like the t-test, the power/required sample size are computed analytically. If you look at the "guts" of a $t$-statistic, it is based on a few things:

• The "effect size" $ES$: in your example, $\overline{(X-6)}$
• The sample size $n$: 100 in your example, but very necessary!
• The sample variance $s_n$: not given in your example
• The significance level/Type I error rate $\alpha$: customarily 0.05

The test works by comparing $$t =\frac{ES\cdot \sqrt{n}}{s_n}$$ to a "critical value"$t_{crit}$ that depends on $\alpha$ and $n$. You could easily rearrange this to solve for $n$ for specific values of the other parameters, and using a bit more math, extend that to calculate power directly.

This chapter by John M. Lachin contains a lot more details.