6
$\begingroup$

I know power is the likelihood of correctly rejecting the null hypothesis, but I just want to check that my interpretation of it with the detectable difference is correct.

Consider a 2 sample test with the null hypothesis that there is no difference between group means.

My understanding is that, by putting a minimum detectable difference of say 5%, with power 80%, into the sample size calculation, then in conjunction with the computed sample size this means that:

if the true difference is 5% then my test will have a likelihood of 80% of rejecting the null hypothesis. My test will also have a slightly higher likelihood of rejecting the null hypothesis if the true difference is 6%, etc.

Is this correct?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Short answer: yes. Longer answer to follow. $\endgroup$
    – Glen_b
    Commented Feb 8, 2014 at 23:50
  • 1
    $\begingroup$ Are you the same person as this user (cf. this question), @Hatti? If so, please register & merge your accounts (see here). This has advantages for you: eg, you will be notified if something happens on 1 of your questions, & your previous good questions have earned you privileges such as the ability to vote. It also helps CV function smoothly & properly. $\endgroup$
    – gung - Reinstate Monica
    Commented Feb 9, 2014 at 0:32

1 Answer 1

Reset to default
7
$\begingroup$

Your understanding looks correct, but when talking to statisticians it's best not to use the word 'likelihood' when you mean 'probability'; in statistics likelihood is a technical term with a very specific meaning (one that isn't the same thing as probability).

When computing sample size, it's common to specify what power is desired for a specific effect size (for some situations, a few additional details or assumptions may be required as well).

If you specify a minimum desired power at a given a sample size, then the effect size that gives that power is often called the "minimum detectable difference".

If the true effect size is larger, the power will be larger (just as you said), if it's smaller, the power will be smaller.

If you draw the relationship between power and effect size, you get a thing called a power curve.

Here's a power curve for a two-sample t-test, n=35 in each group, with the effect size measured in number of $\sigma$'s of difference between means:
enter image description here

The 80% power line is marked in, and corresponds to an effect size of about $0.68\sigma$. In this case both sides of the power curve are shown, but for symmetric cases it's common to omit the left side (or equivalently to put the effect size as $|\mu_1-\mu_2|/\sigma$, thus only getting the curve on right side in the above diagram).

The "power" at effect size 0 is really the probability of rejection when the null is true (i.e. not actually power at that point) - which is of course the significance level, $\alpha$; in the above plot that's 0.05. Normally the rejection rate at that point is drawn in, even though it's not strictly power.

Here are a few other examples of power curves (used in answers to other questions):

(1) A power comparison of paired t-test (curve) and Signed Rank test (points) for 4 pairs of normal observations (it's actually two-sided, but the left half isn't shown as it's a mirror image of the right half):
plot of power for the one-sample t and signed rank test at the normal with n=4

(2) Power for a binomial (one tailed test):
enter image description here

(3) These can even be drawn for goodness of fit tests if you restrict consideration to a particular set of alternatives (in this case, the null is normality, and the subset of alternatives considered in this power comparison are gamma distributions):
power curves for the Shapiro-Wilk and Lilliefors tests of general normality against increasingly skew gamma alternatives

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks so much for your reply, just want to make sure I'm using the correct terminology though, sorry. I think it's because it's called 'minimal' detectable difference. With a 2-sample test, you get an estimate of the difference with a confidence interval. So smaller differences than the MDD can be detected (down to the half-width of the confidence interval) but with smaller power? Thanks again! $\endgroup$
    – Hatti
    Commented Feb 14, 2014 at 15:48
  • $\begingroup$ Yes, that's my understanding of the situation. $\endgroup$
    – Glen_b
    Commented Feb 14, 2014 at 22:36
  • $\begingroup$ Will be also the source code for this analysis provided? Thanks. $\endgroup$
    – Maximilian
    Commented Oct 5, 2018 at 16:25
  • 1
    $\begingroup$ Note that this is a post made more than four and a half years ago. Further, each of the plots contained in it are from posts made earlier still (some may well be 7 or 8 years old), so source code would not be made available here unless I wrote it again. You may be able to locate some of the earlier posts if a particular analysis interests you and some of those might have code. However, I expect it would be much more instructive to write one for yourself. Did you anticipate some difficulty in doing so? If it's not a pure coding-question it may be that you can post an on-topic question here. $\endgroup$
    – Glen_b
    Commented Oct 6, 2018 at 1:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.