How do statistical packages calculate power? For example, suppose we have a sample $X$ of $100$ observations. We assume that they are from a normally distributed population (iid). Our hypothesis test is $H_{0}: \mu = 6$ vs. $H_a: \mu \neq 6$. Suppose the null hypothesis is rejected. How does a software package compute the probability of correctly rejecting the null hypothesis? Wouldn't it have to repeatedly perform the same hypothesis test on different samples? Or is it possible to compute the power by only performing the hypothesis test on one sample?


1 Answer 1


You're correct that one way to perform a power analysis would be via simulation. You could repeatedly generate 100 observations from a distribution and test those observations against the null hypothesis. If it consistently reported a (true) difference, you could say the test has high power.

However, for many common tests, like the t-test, the power/required sample size are computed analytically. If you look at the "guts" of a $t$-statistic, it is based on a few things:

  • The "effect size" $ES$: in your example, $\overline{(X-6)}$
  • The sample size $n$: 100 in your example, but very necessary!
  • The sample variance $s_n$: not given in your example
  • The significance level/Type I error rate $\alpha$: customarily 0.05

The test works by comparing $$t =\frac{ES\cdot \sqrt{n}}{s_n}$$ to a "critical value"$t_{crit}$ that depends on $\alpha$ and $n$. You could easily rearrange this to solve for $n$ for specific values of the other parameters, and using a bit more math, extend that to calculate power directly.

This chapter by John M. Lachin contains a lot more details.

  • $\begingroup$ What happens if you can't apply "common" tests to your data? Then you need to do simulation? Also you mean the effect size is $\overline{X}-6$? $\endgroup$
    – proton
    Jan 21, 2013 at 15:49
  • $\begingroup$ That or run through the math and derive the power calculation yourself. Personally, I find well-done simulations perfectly convincing; I'd just be very surprised if many packages used them for t-tests, ANOVAs, etc. I took a peek inside some R packages for this and they look purely analytical. $\endgroup$ Jan 21, 2013 at 16:01
  • $\begingroup$ +1, nice answer--very clean. I appreciate the reference as well. $\endgroup$ Mar 11, 2013 at 17:06
  • 3
    $\begingroup$ The paper link in Matt Krause's answer seems no longer to work; this one works: ftp.biostat.wisc.edu/pub/chappell/641/papers/paper28.pdf $\endgroup$
    – khstacking
    Oct 2, 2014 at 21:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.