0
$\begingroup$

The critics of this specific study say it was only 80% powered to detect a 50% decrease in the length of hospital stay. I cannot for the life of me figure out how to calculate this using the information available in the paper. Can anyone help me? https://jamanetwork.com/journals/jama/fullarticle/2776738

The study compares (among other things) the length of hospital stay in intervention and control groups. Only medians and interquartile ranges are given in the abstract, maybe any other relevant variables later. Which variables do I need to know anyway? Maybe they’re not all available in the paper.

In my opinion, the study is trustworthy based on its small 95% confidence interval showing the evidence is consistent with the null hypothesis or maybe a small effect (HR 95% CI 0.8-1.4). However, if the critics are right the power to identify even moderate effects must be abysmal...

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

I think they made a mistake in the calculation. Was the criticism published somewhere?

There is not much information to go on in the article. The Statistical Analysis Plan is included in the supplemental content, but it is only a few pages and doesn't have any useful information about that.

I would use these steps:

  1. The observed unadjusted hazard ratio was 1.07.
  2. The observed p-value from the logrank test was 0.59. This implies the observed test statistic was approximately $-\Phi^{-1}(0.59/2) \approx 0.54$.
  3. The logrank test statistic is approximately equal to the log of the observed hazard ratio divided by its standard error. So, the standard error is approximately $\log(1.07)/0.54=0.125$.
  4. The mean should be equal to $\Phi^{-1}(1-0.025) + \Phi^{-1}(0.8) \approx 2.80$ for a one-sided 0.025 test with 80% power.

Combining these together and assuming the standard error remains equal to 0.125, in order to have 80% power we would need the hazard ratio to be $\lambda=e^{2.8 (0.125)}\approx 1.42$. The mean time in hospital (assuming exponential distributions) would need to be decreased by about $1-1/1.42$ or about 30%.

You can verify this using the formula here. The sample size needed assuming the hazard ratio is 1.42 is:
$$\frac{4(z_{0.025}+z_{0.2})^2}{d \log^2 1.42}$$ where $d$ is the probability of having an observed event (in both arms combined). Without any further information, it seems like $d=1$ is a reasonable assumption and then the sample size would be 255.

If I assumed that the hazard ratio was 2, then I would need to use $d=0.27$, i.e. 73% of patients are right censored, to derive a required sample size of 240.

Improve this answer
$\endgroup$
4
  • $\begingroup$ The criticism appeared in an accompanying editorial here: jamanetwork.com/journals/jama/fullarticle/2776736 Quote: “The authors state that the number of participants was chosen on the basis of feasibility, and that with 208 participants they would have 80% power to detect a 50% difference in hospital length of stay, which is a highly improbable result.” $\endgroup$
    – Aerocurios
    Mar 25, 2021 at 17:28
  • $\begingroup$ The authors say the number of participants was chosen on the basis of feasibility in the supplementary material, however, since the calculation never appears in the original text, I assume the critics made the calculation themselves using the information available. Maybe the authors communicated the critics additional information. $\endgroup$
    – Aerocurios
    Mar 25, 2021 at 17:58
  • $\begingroup$ I haven’t done this in a decade and am definitely going to need the dummy version. What is “phi” in “minus phi to the power of minus one multiplied by the p-value divided by two” anyway? And your answer to the question is the study had an 80% power to detect a 30% difference it appears? I’m going to need open my textbook again. $\endgroup$
    – Aerocurios
    Mar 25, 2021 at 20:16
  • $\begingroup$ $\Phi(x)$ is the standard normal distribution function. For example, the upper 0.025 quantile is $z_{0.025}=1.96$, so $\Phi(1.96)=0.025$. Yes, with a sample size of 240 and the other assumptions (7 days median hospitalization, no censoring), the study had 80% power to find a 30% reduction in days of hospitalization. $\endgroup$
    – John L
    Mar 26, 2021 at 0:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.