I'm reading this paper http://circ.ahajournals.org/content/134/25/2095 , with the aim of applying the same method to my data analysis.

In the paper, the authors compared 30-day survival chances of cardiac-arrest patients with or without bystander cardiopulmonary resuscitation (CPR). (see Fig. 2 attached)

They used a multiple logistic regression model taking many factors into account, and the relationship between 30-day survival chances and ambulance response time was modeled by restricted cubic splines with prespecified knots at 5, 10, 15, 20 min.

enter image description here

After drawing this graph, they abruptly concluded that "The association of between 30-day survival and bystander CPR compared with no-bystander CPR became statistically insignificant when response time was more than 13 minutes (3.7% [95% CI: 2.2-5.4] for bystander CPR vs 1.5% [95% CI: 0.6-2.7] for no-bystander CPR)", with no p value shown. (They write in the Methods section that CI is a 95% bootstrap confidence interval based on 2000 bootstrap samples.)

How did they determine that the difference between the two groups is "statistically insignificant" after 13 min?

  • 3
    $\begingroup$ My guess is that after that time confidence intervals for both curves are overlapping, this means there's a probability that the both values are equal (difference could be insignificant). $\endgroup$ – Maju116 Mar 10 '17 at 11:04
  • 1
    $\begingroup$ I agree with @Maju116. While 95% CI and p-value are not always equivalent, when the CI obviously overlap, the predicted probabilities of survival of the two groups became statistically indistinguishable at the 95% level. One can always run an F-test to see whether they are really different prior to 13 mins and not that different afterwards. $\endgroup$ – Carl PCH Mar 16 '17 at 14:45

This is not really a complete answer - more like a too-long comment - but it may address some of the uncertainties you have about the authors' approach. Firstly, I'm fairly certain that the approach described by @Maju116 is the essence of what was done. It seems good enough to me to gain a quick, visual understanding of the associations at play and the strength of differences between groups, but it's inappropriate to claim statistical 'insignificance' in the classical sense.

The claim of a 'statistically insignificant' difference at 13 minutes implies a significance test was performed, with a null hypothesis of "no difference at 13 mins". Clearly this wasn't a pre-specified hypothesis made at the start of the study. And there's no mention of any significance test that assessed this anyway. So it isn't insignificance in the usual sense of being derived from a calculated p-value and compared to a threshold, and therefore it's quite misleading. There's also the implication that a similar comparison has been made at every minute, so there's a multiple-comparisons problem that hasn't been accounted for.

In reality, significance testing is not really an appropriate framework to use here. The null hypothesis would be something like 'no curve difference between groups', but there are so many different ways to define and test for a difference in curves (especially curves that haven't been defined parametrically), that it's hopelessly vulnerable to 'researcher degrees-of-freedom' (see http://andrewgelman.com/2012/11/01/researcher-degrees-of-freedom/). Better just to be completely transparent about how the curves were made, plot them both with uncertainty banding, and discuss how it looks. Or better still, let the reader decide.

In summary, it looks to me like there is some residual differences between the groups after accounting for modelled confounders, such that bystander CPR is associated with a survival benefit if the ambulance arrives within about 13 minutes. No need to complicate this narrative with a spurious significance test.

Two other points:

  1. It's not clear whether they bootstrapped the data and generated curves from each model (good), or made a single model on the whole data then applied it to the bootstrapped data (not good).
  2. The curves aren't smooth enough for them to have used cubic-splines in this figure. It seems they've just independently calculated standardized 30-day survival in each group at each minute up to 20, then again at 30 (curve between 20 and 30 is completely straight). Nothing wrong with this in principle.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.