I am recently using bootstrap for statistical inference and confidence interval building in the setting of regression, especially logistic regression. In many works I've been doing I find that using non parametric bootstrapping compared to classical likelihood based analytic methods tends to give smaller (and therefore boost significance) confidence intervals when the outcome or the predictor is a rare event.

As an example I show some data from a current study we're doing.

We have the following relationship between the principal predictor, ART.type (a multimodal categorical variable with FIVET.ICSI as baseline) and the outcome, thrombocytopenia (a binary variable):

                 thrombocytopenia  no   yes
FIVET.ICSI                         366  9
Ovodonation                        37   0
Stimulation.IUI                    33   3

total number of observation is 448. It's relevant to note that there are any thrombocytopenia cases in women which used Ovodonation.

Using a logistic regression model with a Cauchy prior for the coefficient (preferable in the case of rare events or quasi perfect separation, [ http://www.stat.columbia.edu/~gelman/research/published/priors11.pdf ]) I get the following adjusted ORs (there are other covariates in the model, not shown in the results)

> bayesglm(thrombocytopenia ~ ART.type + age + smoke + nulliparity + syst.disease + chorioniocity, binomial, Data.mothers) %>% confint.default() %>% exp()

                            2.5 %   97.5 %
ART.type.Ovodonation        0.013   4.92
ART.type.Stimulation.IUI    0.94    14.1

Even if Stimulation.IUI is at the edge of significance with a positive effect it doesn't cross the OR = 1 line.

Same analysis but with non-parametric bootstrapping. The reported 95% CI are the bootstrap accelerated, bias adjusted confidence intervals [DiCiccio, T.J. and Efron B. (1996) Bootstrap confidence intervals. Statistical Science, 11, 189–228] made with 10000 resamplings.

                            2.5 %   97.5 %
ART.type.Ovodonation        0.121   0.423
ART.type.Stimulation.IUI    0.547   18.5

In this analysis Ovodonation become significant while Stimulation.IUI gets large confidence intervals.

I think that analytic methods for logistic regression have problems with situation in which you have zero cases for a class. My idea is that the bootstrap results make sense because if you reason proportionally, as logistic regression does, 0/37 = 0% is hugely less than 9/366 = 2.5% and 3/33 = 9% is way more, so Ovodonation can be justified as associated negatively with thrombocytopenia while Stimulation.IUI positively, compared to FIVET.ICSI. But there is an inherent instability in such results that make the interpretation troublesome; in facts, it would be sufficient even one more case in the Ovodonation to have odds of 1/37 = 2.7% and therefore slightly more than the odds of FIVET.ICSI, reverting the result. One actually cannot know whether we are observing a causal strong protective effect or a lucky case.

But instability of results and real causality is different from classical lack of significance, so I think that, given the data, bootstrap results are more correct from this point of view. Given the data the presence of an effect is undeniable.

I want to know if my thesis is wrong and significance with bootstrap in this cases is more likely a type 1 error; or, if it is the classical analysis that suffers from type 2 error, how do I justify (maybe with references) significance for a such aleatory result to an audience (medical doctors) new to the concept of boostrap and which usually misinterpret statistical significance.

  • 1
    $\begingroup$ Link to the Gelman article is broken... $\endgroup$ – Mike Hunter Mar 11 '16 at 10:58
  • $\begingroup$ @DJohnson stackexchange parser was including the ] in the link. $\endgroup$ – Bakaburg Mar 11 '16 at 11:15
  • 1
    $\begingroup$ @Tim I see reference to general failing cases for bootstrap in the question you linked, but nothing about my specific case. Or did i miss something? $\endgroup$ – Bakaburg Mar 11 '16 at 11:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.