I want to calculate a bootstrap confidence interval for a mean difference between two groups. Apparently, there are two ways to do it, depending on whether the null hypothesis of a zero difference between the groups is accepted or not.

Here are my approaches:

If the null hypothesis is assumed to be true, the bootstrap distribution of mean differences stems from merging the values of the two groups, randomly reassigning group memberships (in order to obtain a distribution under the null), and calculating the test-statistic.

If the null hypothesis is assumed to be false, I do not merge the values of the groups but sample separately from a each group.

Since I observe quite substantial differences for the two approaches, I wonder whether there is some guideline when to use which.

Thanks for your aid,


The first approach is a randomisation test not a bootstrap test. Randomisation tests require 'exchangeability'. This is the assumption that the underlying distributions in the two samples are the same, only the means are different (see P.I Good, "Resampling Methods - A Practical Guide to Data Analysis", 3rd Edition, 2006). If the sample distributions are quite different, this may explain why your results are inconsistent.

  • $\begingroup$ +1 Very good points. My friend Phil Good would be happy to see the reference to him. Many years ago he and I taught a short course on resampling. I did the bootstrap and he did the permutation tests. His first book published by Springer in the 1990s was strictly on permutation methods. $\endgroup$ – Michael R. Chernick Jan 12 '17 at 5:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.