Assuming two samples of numeric values for two groups of unequal group sizes (e.g. 100 opinion scores collected from group A and 15 opinion scores collected from group B), I understand that non-overlapping 95% confidence intervals of the opinion scores indicate that there is a statistically significant difference in the scores of the two groups, while the opposite is not necessarily true: overlapping confidence intervals do not necessarily indicate the lack of statistically significant difference (see e.g. https://www.cscu.cornell.edu/news/statnews/stnews73.pdf).

My question is: What if the confidence intervals are computed not from the observed opinion scores, but by bootstrapping mean scores for both groups? Do overlapping confidence intervals obtained from bootstrapping rather than from the observed scores also indicate that the difference might be statistically significant despite the confidence interval overlap? Or does an overlap of bootstrapped confidence intervals necessarily mean that there is no statistically significant difference between the two groups?

My own attempt to answer this question: Given that rejecting statistical significance based on confidence intervals is only meaningful when the confidence intervals are computed on the differences between groups rather than group means and if the confidence interval of differences does not include zero (see e.g. https://statisticsbyjim.com/hypothesis-testing/confidence-intervals-compare-means), I conclude that bootstrapping in itself does not necessarily mean that statistical significance can be rejected if bootstrapped confidence intervals from group means are non-overlapping. Is my thinking correct? Or does bootstrapping from the two samples allows to safely assume lack of statistical significance when confidence intervals overlap?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.