Question regarding post-hoc power analysis I am having trouble with interpretation of a prospective superiority randomised controlled trial. Study characteristics:
Study design:

*

*Alpha value = 0.05

*1-beta = 0.8

*Predicted effect size =0.15 [P1 = 0.65, P2 = 0.5]

*Numbers needed = 492

Observed results:

*

*Effect size = 0.09 [P1 = 0.50, P2 = 0.41]


*Numbers enrolled = 502


*statistically significant P value -> P = 0.03


*null hypothesis is rejected.
My impression is that the trial underestimated the effect size. If I use the observed results to calculate a sample size that would take into the effect size (same alpha and beta), I get a numbers needed of 900+ patients. This would suggest that the trial is actually under powered. And potentially, the result we see is a false positive result.
 A: "My impression is that the trial underestimated the effect size." The only reason to believe this is if you have good prior information such as existing prior data that indicates that the effect should be larger. The computed numbers give no indication whatsoever that the effect size is underestimated, or rather, it may be underestimated as well as overestimated due to standard statistical variation. "And potentially, the result we see is a false positive result." - This is always a possibility with p=0.03. Once more, the post hoc power analysis is not informative about this.
A: Generally speaking, post-hoc power analysis should be avoided. One of the key problems is that the observed effect size is estimated with a great deal of uncertainty. For the case you present here, the observed effect size is 0.09. The standard error of this estimate is given as follows:
$$ \sqrt{\frac{p_t \times (1 - p_t)} {n_t} + \frac{p_c \times (1 - p_c)}{n_c} } $$
Assuming the sample was evenly split between the treatment and control, this gives us a standard error of 0.044.
sqrt( ((0.5 * (1 - 0.5)) / 251) + ((0.41 * (1 - 0.41)) / 251))

That gives us a 95 percent confidence interval of 0.003 to 0.177. This is of course compatible with the predicted effect size of 0.15. It's also compatible with an effect size of almost zero. There's too much uncertainty about the observed effect size to make it useful for post-hoc power analysis.
