5
$\begingroup$

My PI is having me run a post-hoc power analysis on our samples, with a treatment and control group and a binary outcome. I was using this website to start the process. However, at the end, the website cautions:

The Dangers of Post-Hoc Analysis

Post-hoc power analysis has been criticized as a means of interpreting negative study results.2 Because post-hoc analyses are typically only calculated on negative trials (p ≥ 0.05), such an analysis will produce a low post-hoc power result, which may be misinterpreted as the trial having inadequate power.

As an alternative to post-hoc power, analysis of the width and magnitude of the 95% confidence interval (95% CI) may be a more appropriate method of determining statistical power.

What does this mean? How would I use the width of the CI to determine if the statistical power was adequate? Is there a common threshold that I should consider?

Improve this question
$\endgroup$
3

1 Answer 1

Reset to default
7
$\begingroup$

If your CIs are narrow, then you have an idea of how large the effect is, and you can say with some confidence that the effect is small, and that's why you didn't detect it.

If the CIs are wide, then you don't know how big the effect is. Maybe it's big and you didn't detect it because you didn't have enough power. Maybe it's small and you didn't detect it because it's small.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you, this makes so much more sense $\endgroup$
    – tchoup
    Jun 17, 2022 at 20:55
  • $\begingroup$ Follow up question if you have the time, how do you think about "narrow". I have one variable that is non-significant, it has a coefficient of -0.4, and a CI of -2.2:0.6, and so that seems small overall, but kind of big compared to the coefficient. Or should I think of the overall variation within the variable itself? thanks! $\endgroup$
    – tchoup
    Jun 17, 2022 at 23:50
  • 1
    $\begingroup$ I think that narrow depends on the interpretation of the measure. If you're talking about people's heights, and you're within a half inch, that's close. If you're talking about the length of mice, that's big. How big is -2.2? So small you don't care? You had plenty of power. So big that that's a big effect? You didn't have enough power. $\endgroup$
    – Jeremy Miles
    Jun 18, 2022 at 4:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.