# How statistical packages calculate power

How do statistical packages calculate power? For example, suppose we have a sample $X$ of $100$ observations. We assume that they are from a normally distributed population (iid). Our hypothesis test is $H_{0}: \mu = 6$ vs. $H_a: \mu \neq 6$. Suppose the null hypothesis is rejected. How does a software package compute the probability of correctly rejecting the null hypothesis? Wouldn't it have to repeatedly perform the same hypothesis test on different samples? Or is it possible to compute the power by only performing the hypothesis test on one sample?

-

You're correct that one way to perform a power analysis would be via simulation. You could repeatedly generate 100 observations from a distribution and test those observations against the null hypothesis. If it consistently reported a (true) difference, you could say the test has high power.

However, for many common tests, like the t-test, the power/required sample size are computed analytically. If you look at the "guts" of a $t$-statistic, it is based on a few things:

• The "effect size" $ES$: in your example, $\overline{(X-6)}$
• The sample size $n$: 100 in your example, but very necessary!
• The sample variance $s_n$: not given in your example
• The significance level/Type I error rate $\alpha$: customarily 0.05

The test works by comparing $$t =\frac{ES\cdot \sqrt{n}}{s_n}$$ to a "critical value"$t_{crit}$ that depends on $\alpha$ and $n$. You could easily rearrange this to solve for $n$ for specific values of the other parameters, and using a bit more math, extend that to calculate power directly.

This chapter by John M. Lachin contains a lot more details.

-
What happens if you can't apply "common" tests to your data? Then you need to do simulation? Also you mean the effect size is $\overline{X}-6$? –  proton Jan 21 '13 at 15:49
That or run through the math and derive the power calculation yourself. Personally, I find well-done simulations perfectly convincing; I'd just be very surprised if many packages used them for t-tests, ANOVAs, etc. I took a peek inside some R packages for this and they look purely analytical. –  Matt Krause Jan 21 '13 at 16:01
+1, nice answer--very clean. I appreciate the reference as well. –  gung Mar 11 '13 at 17:06
The paper link in Matt Krause's answer seems no longer to work; this one works: ftp.biostat.wisc.edu/pub/chappell/641/papers/paper28.pdf –  khstacking Oct 2 '14 at 21:27
Thanks, @khstacking! I updated the link. –  Matt Krause Oct 2 '14 at 21:56