It is well known that a study with low statistical power has a reduced chance of detecting a true effect. But if low power also reduces the likelihood that a statistically significant result reflects a true effect (as shown in Power failure: why small sample size undermines the reliability of neuroscience), why does one report statistical power only when results are non significant?

As I can't comment yet, I edit my question in response to @Glen_b and @Penguin_Knight : I should have said "discuss" instead of "report". Of course, one should do the power calculation a priori. But my question is really: Why people who get small p-value with low statistical power don't moderate their findings?

  • $\begingroup$ Good quality research should report prospective power calculations when describing methods irrespective of the outcome. E.g., CONSORT statement for randomized controlled trials in medicine includes sample size calculation methods as an item to report. $\endgroup$
    – tristan
    Feb 25 '14 at 13:27
  • 1
    $\begingroup$ "why does one report statistical power only when results are non significant?" -- that's a new one on me. You would do the calculation before you know whether the result is significant. Who says to leave it out if the result is significant? $\endgroup$
    – Glen_b
    Feb 25 '14 at 13:29
  • $\begingroup$ @Glen_b, I haven't seen writing guideline asking people to write so, but I have seen journal critique guidelines asking readers to look for power calculation for negative trials. $\endgroup$ Feb 25 '14 at 13:39
  • $\begingroup$ Thanks for the clarification. I read your edited question and I don't feel my answer would work you. I'd delete it and hopefully someone can fill in. $\endgroup$ Feb 25 '14 at 14:10

If the result is not statistically significant, there are two possibilities. One is that the null hypothesis is true. The other is that the null hypothesis is false (so there really is a difference between the populations) but some combination of small sample size, large scatter and bad luck led your experiment to a conclusion that the result is not statistically significant.

Running a power analysis can help understand the results. A power calculation answers this question:

If the true difference between populations is a stated hypothetical value (a value you would find large enough to be worth detecting), what is the chance that a study of the size you just did (given the scatter you observed) would conclude that the difference is statistically significant?

Interpreting the results:

  • If the power to detect the difference you would have cared about is high, then your results are pretty good evidence that the actual difference is likely to be smaller than your hypothetical value. You have solid negative data.
  • If the power to detect that difference is low, then you really can't conclude much from your data. Your findings are equivocal.

Hopefully, the explanation above shows how a power analysis can be helpful in interpreting a not-statistically-significant result. In contrast, power analyses don't help much when the result is statistically significant.

Important note: The power analysis should be set to compute the power to detect the smallest difference that you would find scientifically (or clinically) worth detecting. It is not even a tiny bit helpful to run a power analysis set to compute the power to detect the difference that your study actually detected. Such post-hoc or observed power calculations are invalid if they are based on the effect actually observed.

2019 UPDATE. While I think everything above is true, I am not sure it is so helpful. Much better to compute and interpret the 95% confidence interval for the difference (or ratio) and not even think about power. Power really is a way to quantify the effectiveness of a proposed experiment and not a good way to quantify or understand the results of a completed experiment...

  • $\begingroup$ "power analyses don't help much when the result is statistically significant" Can you comment of the situation when we have p<0.05 or whatever but the clinical significance is small. How can we interpret that apart from "probably not worth paying attention because other things have larger effect" $\endgroup$ Jul 18 '19 at 17:41
  • 1
    $\begingroup$ @aaaaaa I think your interpretation is correct. I just updated my answer to point out that looking at confidence intervals is a much better way to interpret data than computing power after the fact. $\endgroup$ Jul 19 '19 at 21:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.