I am recently using bootstrap for statistical inference and confidence interval building in the setting of regression, especially logistic regression. In many works I've been doing I find that using non parametric bootstrapping compared to classical likelihood based analytic methods tends to give smaller (and therefore boost significance) confidence intervals when the outcome or the predictor is a rare event.
As an example I show some data from a current study we're doing.
We have the following relationship between the principal predictor, ART.type (a multimodal categorical variable with FIVET.ICSI as baseline) and the outcome, thrombocytopenia (a binary variable):
thrombocytopenia no yes ART.type FIVET.ICSI 366 9 Ovodonation 37 0 Stimulation.IUI 33 3
total number of observation is 448. It's relevant to note that there are any thrombocytopenia cases in women which used Ovodonation.
Using a logistic regression model with a Cauchy prior for the coefficient (preferable in the case of rare events or quasi perfect separation, [ http://www.stat.columbia.edu/~gelman/research/published/priors11.pdf ]) I get the following adjusted ORs (there are other covariates in the model, not shown in the results)
> bayesglm(thrombocytopenia ~ ART.type + age + smoke + nulliparity + syst.disease + chorioniocity, binomial, Data.mothers) %>% confint.default() %>% exp() 2.5 % 97.5 % ART.type.Ovodonation 0.013 4.92 ART.type.Stimulation.IUI 0.94 14.1
Even if Stimulation.IUI is at the edge of significance with a positive effect it doesn't cross the OR = 1 line.
Same analysis but with non-parametric bootstrapping. The reported 95% CI are the bootstrap accelerated, bias adjusted confidence intervals [DiCiccio, T.J. and Efron B. (1996) Bootstrap confidence intervals. Statistical Science, 11, 189–228] made with 10000 resamplings.
2.5 % 97.5 % ART.type.Ovodonation 0.121 0.423 ART.type.Stimulation.IUI 0.547 18.5
In this analysis Ovodonation become significant while Stimulation.IUI gets large confidence intervals.
I think that analytic methods for logistic regression have problems with situation in which you have zero cases for a class. My idea is that the bootstrap results make sense because if you reason proportionally, as logistic regression does, 0/37 = 0% is hugely less than 9/366 = 2.5% and 3/33 = 9% is way more, so Ovodonation can be justified as associated negatively with thrombocytopenia while Stimulation.IUI positively, compared to FIVET.ICSI. But there is an inherent instability in such results that make the interpretation troublesome; in facts, it would be sufficient even one more case in the Ovodonation to have odds of 1/37 = 2.7% and therefore slightly more than the odds of FIVET.ICSI, reverting the result. One actually cannot know whether we are observing a causal strong protective effect or a lucky case.
But instability of results and real causality is different from classical lack of significance, so I think that, given the data, bootstrap results are more correct from this point of view. Given the data the presence of an effect is undeniable.
I want to know if my thesis is wrong and significance with bootstrap in this cases is more likely a type 1 error; or, if it is the classical analysis that suffers from type 2 error, how do I justify (maybe with references) significance for a such aleatory result to an audience (medical doctors) new to the concept of boostrap and which usually misinterpret statistical significance.