6
$\begingroup$

It is well known that a study with low statistical power has a reduced chance of detecting a true effect. But if low power also reduces the likelihood that a statistically significant result reflects a true effect (as shown in Power failure: why small sample size undermines the reliability of neuroscience), why does one report statistical power only when results are non significant?

As I can't comment yet, I edit my question in response to @Glen_b and @Penguin_Knight : I should have said "discuss" instead of "report". Of course, one should do the power calculation a priori. But my question is really: Why people who get small p-value with low statistical power don't moderate their findings?

$\endgroup$
4
  • 1
    $\begingroup$ Good quality research should report prospective power calculations when describing methods irrespective of the outcome. E.g., CONSORT statement for randomized controlled trials in medicine includes sample size calculation methods as an item to report. $\endgroup$
    – tristan
    Commented Feb 25, 2014 at 13:27
  • 4
    $\begingroup$ "why does one report statistical power only when results are non significant?" -- that's a new one on me. You would do the calculation before you know whether the result is significant. Who says to leave it out if the result is significant? $\endgroup$
    – Glen_b
    Commented Feb 25, 2014 at 13:29
  • $\begingroup$ @Glen_b, I haven't seen writing guideline asking people to write so, but I have seen journal critique guidelines asking readers to look for power calculation for negative trials. $\endgroup$ Commented Feb 25, 2014 at 13:39
  • $\begingroup$ Thanks for the clarification. I read your edited question and I don't feel my answer would work you. I'd delete it and hopefully someone can fill in. $\endgroup$ Commented Feb 25, 2014 at 14:10

2 Answers 2

12
$\begingroup$

If the result is not statistically significant, there are two possibilities. One is that the null hypothesis is true. The other is that the null hypothesis is false (so there really is a difference between the populations) but some combination of small sample size, large scatter and bad luck led your experiment to a conclusion that the result is not statistically significant.

Running a power analysis can help understand the results. A power calculation answers this question:

If the true difference between populations is a stated hypothetical value (a value you would find large enough to be worth detecting), what is the chance that a study of the size you just did (given the scatter you observed) would conclude that the difference is statistically significant?

Interpreting the results:

  • If the power to detect the difference you would have cared about is high, then your results are pretty good evidence that the actual difference is likely to be smaller than your hypothetical value. You have solid negative data.
  • If the power to detect that difference is low, then you really can't conclude much from your data. Your findings are equivocal.

Hopefully, the explanation above shows how a power analysis can be helpful in interpreting a not-statistically-significant result. In contrast, power analyses don't help much when the result is statistically significant.

Important note: The power analysis should be set to compute the power to detect the smallest difference that you would find scientifically (or clinically) worth detecting. It is not even a tiny bit helpful to run a power analysis set to compute the power to detect the difference that your study actually detected. Such post-hoc or observed power calculations are invalid if they are based on the effect actually observed.

2019 UPDATE. While I think everything above is true, I am not sure it is so helpful. Much better to compute and interpret the 95% confidence interval for the difference (or ratio) and not even think about power. Power really is a way to quantify the effectiveness of a proposed experiment and not a good way to quantify or understand the results of a completed experiment...

$\endgroup$
2
  • $\begingroup$ "power analyses don't help much when the result is statistically significant" Can you comment of the situation when we have p<0.05 or whatever but the clinical significance is small. How can we interpret that apart from "probably not worth paying attention because other things have larger effect" $\endgroup$ Commented Jul 18, 2019 at 17:41
  • 1
    $\begingroup$ @aaaaaa I think your interpretation is correct. I just updated my answer to point out that looking at confidence intervals is a much better way to interpret data than computing power after the fact. $\endgroup$ Commented Jul 19, 2019 at 21:34
3
$\begingroup$

Reporting power only when the results are non-significant is a misunderstanding of the purpose of a power calculation and is a poor practice that was highlighted in the question. Before the study was conducted, the researcher should have a good idea of the statistical power of the sample size they selected for the effect of interest. Reporting that pre-experiment power is something that should be done more often.

Two other poor practices that go along with this observation:

  1. The researchers who only report power with non-significant results, are trying to justify their non-significant result because they are sure that there should be an effect and something "went wrong" with the study.
  2. Some of these researchers are calculating post-hoc power. Once they learn the actual observed variance, they want to adjust their computed power estimate. There are many commentators who can explain why this is a bad idea 1, 2.

Possible reasons that some researchers report power only when a result is non-significant:

  1. The targeted journal wants to be parsimonious and feels power is irrelevant for significant results.
  2. Researcher wants to justify a study that doesn't observe the effect that they believed should be present.
  3. Misunderstanding of power when calculating post-hoc power.

Possible reasons why researchers don't moderate small p-values by reporting low power.

  1. Readers might ask why they undertook the statistical analysis in the first place when the effect they were looking for was not observable with good power.
  2. It would undermine the conclusion they are trying to draw from the significant result.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.