1
$\begingroup$

I am having trouble with interpretation of a prospective superiority randomised controlled trial. Study characteristics:

Study design:

  • Alpha value = 0.05
  • 1-beta = 0.8
  • Predicted effect size =0.15 [P1 = 0.65, P2 = 0.5]
  • Numbers needed = 492

Observed results:

  • Effect size = 0.09 [P1 = 0.50, P2 = 0.41]

  • Numbers enrolled = 502

  • statistically significant P value -> P = 0.03

  • null hypothesis is rejected.

My impression is that the trial underestimated the effect size. If I use the observed results to calculate a sample size that would take into the effect size (same alpha and beta), I get a numbers needed of 900+ patients. This would suggest that the trial is actually under powered. And potentially, the result we see is a false positive result.

Improve this question
$\endgroup$
1
  • $\begingroup$ Post-hoc power analysis is a transformation of the p-value, per this paper and many others $\endgroup$
    – alan ocallaghan
    Commented Apr 13, 2022 at 13:01

2 Answers 2

Reset to default
1
$\begingroup$

"My impression is that the trial underestimated the effect size." The only reason to believe this is if you have good prior information such as existing prior data that indicates that the effect should be larger. The computed numbers give no indication whatsoever that the effect size is underestimated, or rather, it may be underestimated as well as overestimated due to standard statistical variation. "And potentially, the result we see is a false positive result." - This is always a possibility with p=0.03. Once more, the post hoc power analysis is not informative about this.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thank you Christian, very helpful. I have a few follow up questions. If previous pilot studies suggested an effect size of 0.15, and we get an observed effect size of 0.09, does that imply understimation of effect size? I am trying to contextualise the concept of "delta inflation" doi: 10.1186/cc8990 Should we be concerned about "delta inflation" only if the null hypothesis cannot be rejected? Would it not make sense to do an interm analysis, and sample adjust based on the lower estimated effect size? doi.org/10.1177%2F009286159302700317 $\endgroup$
    – NKJHH
    Commented Apr 14, 2022 at 2:21
  • $\begingroup$ "If previous pilot studies suggested an effect size of 0.15, and we get an observed effect size of 0.09, does that imply understimation of effect size?" Not necessarily but with some probability. Ultimately it depends on quality, precision, and comparability of the previous studies (which could be taken into account in a Bayesian approach). Can't comment on "delta inflation" as I'm not an expert. Interim analysis looks OK but I haven't looked into this in detail. For planning your study you better do a proper collaboration with an expert rather than relying on free internet help. $\endgroup$
    – Christian Hennig
    Commented Apr 14, 2022 at 9:17
  • $\begingroup$ Ah I see. I'm not running any trials at all. Just self teaching. Not many experts around unfortunately. But I really do appreciate your input. $\endgroup$
    – NKJHH
    Commented Apr 14, 2022 at 9:23
  • $\begingroup$ I had a further look at "delta inflation". This basically means that there is a tendency to choose the effect size on which a study size is based larger than the effect really is. If it's your own study obviously you are responsible for this yourself. Very often this is not specified based on prior studies, but rather follows a different logic, see below (to be continued). $\endgroup$
    – Christian Hennig
    Commented Apr 14, 2022 at 9:39
  • $\begingroup$ One problem I have with this concept is that often the "predicted" effect size is not so much a prediction but rather a specification of a meaningful and relevant effect size in the context, potentially taking into account costs, side effects, and the like. In which case the "predicted delta", even if larger than the estimated one, isn't "inflated", and an insignificant result isn't a problem as the potentially true smaller effect is not seen as justifying the costs. The "delta inflation" concept comes from the idea that significance is always desirable, but this isn't necessarily the case. $\endgroup$
    – Christian Hennig
    Commented Apr 14, 2022 at 9:41
0
$\begingroup$

Generally speaking, post-hoc power analysis should be avoided. One of the key problems is that the observed effect size is estimated with a great deal of uncertainty. For the case you present here, the observed effect size is 0.09. The standard error of this estimate is given as follows:

$$ \sqrt{\frac{p_t \times (1 - p_t)} {n_t} + \frac{p_c \times (1 - p_c)}{n_c} } $$

Assuming the sample was evenly split between the treatment and control, this gives us a standard error of 0.044.

sqrt( ((0.5 * (1 - 0.5)) / 251) + ((0.41 * (1 - 0.41)) / 251))

That gives us a 95 percent confidence interval of 0.003 to 0.177. This is of course compatible with the predicted effect size of 0.15. It's also compatible with an effect size of almost zero. There's too much uncertainty about the observed effect size to make it useful for post-hoc power analysis.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you num39 $\endgroup$
    – NKJHH
    Commented Apr 14, 2022 at 9:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.