Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

How do statistical packages calculate power? For example, suppose we have a sample $X$ of $100$ observations. We assume that they are from a normally distributed population (iid). Our hypothesis test is $H_{0}: \mu = 6$ vs. $H_a: \mu \neq 6$. Suppose the null hypothesis is rejected. How does a software package compute the probability of correctly rejecting the null hypothesis? Wouldn't it have to repeatedly perform the same hypothesis test on different samples? Or is it possible to compute the power by only performing the hypothesis test on one sample?

share|improve this question
add comment

1 Answer

up vote 4 down vote accepted

You're correct that one way to perform a power analysis would be via simulation. You could repeatedly generate 100 observations from a distribution and test those observations against the null hypothesis. If it consistently reported a (true) difference, you could say the test has high power.

However, for many common tests, like the t-test, the power/required sample size are computed analytically. If you look at the "guts" of a $t$-statistic, it is based on a few things:

  • The "effect size" $ES$: in your example, $\overline{(X-6)}$
  • The sample size $n$: 100 in your example, but very necessary!
  • The sample variance $s_n$: not given in your example
  • The significance level/Type I error rate $\alpha$: customarily 0.05

The test works by comparing $$t =\frac{ES\cdot \sqrt{n}}{s_n}$$ to a "critical value"$t_{crit}$ that depends on $\alpha$ and $n$. You could easily rearrange this to solve for $n$ for specific values of the other parameters, and using a bit more math, extend that to calculate power directly.

This chapter by John M. Lachin contains a lot more details.

share|improve this answer
    
What happens if you can't apply "common" tests to your data? Then you need to do simulation? Also you mean the effect size is $\overline{X}-6$? –  proton Jan 21 '13 at 15:49
    
That or run through the math and derive the power calculation yourself. Personally, I find well-done simulations perfectly convincing; I'd just be very surprised if many packages used them for t-tests, ANOVAs, etc. I took a peek inside some R packages for this and they look purely analytical. –  Matt Krause Jan 21 '13 at 16:01
    
+1, nice answer--very clean. I appreciate the reference as well. –  gung Mar 11 '13 at 17:06
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.