Scenario: I have two measurement tools A and B and only about n=5-10 measurements of the same object for each tool.

I want to test if there is a difference in the mean of the measurements between the two tools. I want to use the bootstrap hypothesis approach creating 1000 bootstrapped samples with replacement for each tool (or do I need to take the values of both tools together and create bootstrap samples?). Here I get 1000 estimated means and a confidence interval from the bootstrapped sample distribution.

Two separate questions:

1.) Is it possible to use these 1000 means for each tool as "values" in a non-parametric test (for example a Mann-Whitney-U-test)? I was wondering if this "upsampling" of measurement points is valid?

2.) Is it possible to use the estimated mean and CI of the bootstrap sampling distribution and if the confidence intervals of both groups don't overlap, can I conclude that they are significantly different?

Thank you.

  • 1
    $\begingroup$ (1) Seems a bad idea. If you're MWW 2-sample test, just use original data. And remember: re-samples are an analytic device; they don't add info and don't count as real world data. (2) There are ways to convert bootstrap CIs to tests. You can google that. I prefer to use permutation tests--see example in my Answer. $\endgroup$
    – BruceET
    Commented May 4, 2020 at 7:38
  • $\begingroup$ Thank you for your excellent answer. Staying with the bootstrap approach though: I have watched this video: youtu.be/9STZ7MxkNVg . If I understand this video and the R example (youtu.be/Zet-qmEEfCU) correctly the bootstrapped datasets are created taking all measurements of both groups together and afterwards calculating the difference in the means of the two groups of each bootstrap which is then compared to the observed difference (the observed test statistic). If I understand it correctly it is very similar to the permutation approach?! $\endgroup$ Commented May 4, 2020 at 18:31
  • $\begingroup$ There are many styles of bootstrap--even in the simplest case where there is only one sample. So you could see a wide variety of approaches on YouTube and elsewhere on the Internet, perhaps not all of them optimal. // I prefer not to comment on, or encourage, bootstrap re-sampling from samples as small as yours. $\endgroup$
    – BruceET
    Commented May 4, 2020 at 20:28
  • $\begingroup$ Ok, but you would encourage permutation testing in such a small sample? I have that permutation testing doesn't give you a confidence interval?! $\endgroup$ Commented May 5, 2020 at 6:54
  • $\begingroup$ Of course, generally speaking, larger samples give better power. But permutation distributions for one-sample tests can give useful results for samples as small as five $(1/2^5 < 5\%)$, and for two-sample tests with each sample as small as four $(1/{8\choose 4} < 5\%).$ // There is a duality btw tests and CIs. A CI can be viewed as a collection of non-rejectable null values of a parameter. So permutation tests can yield CIs and bootstrap CIs can be used to test hypotheses. $\endgroup$
    – BruceET
    Commented May 5, 2020 at 7:09

1 Answer 1


Suppose you have 7 subjects (or objects) and have measures A and B on each, with (fake) data as follows:

Subject      1      2      3      4      5      6      7
A        77.04 111.79 109.82  90.02  97.93  84.01 105.72
B        82.80 117.28 109.58  96.13 100.72  88.07 111.46
D = B-A   5.76   5.49  -0.24   6.11   2.79   4.06   5.74

A paired t test (one-sample t test on difference) shows a significant difference at the 0.3% level, as follows:

d = c(5.76, 5.49, -0.24, 6.11, 2.79, 4.06, 5.74)
[1] 0.002778031

But suppose you doubt the normality of the data and worry about using a t test for such a small sample.

A nonparametric permutation test is based on a large number of random permutations of the sign of the differences. For each permutation the mean difference is found. (One says that the difference is the 'metric' for the permutation test.) The observed difference is $\bar D_{obs} = 4.244.$

[1] 4.244286

The P-value of the two-sided permutation test is the proportion of permuted differences that equals or exceeds the observed difference in absolute value. In this case, the P-value is 0.03.

a.prm = replicate(10^5, mean(sample(c(-1,1),7,rep=T)*d))
mean(abs(a.prm) >= abs(mean(d)))
[1] 0.03029

With some trouble one could obtain the exact permutation distribution by combinatorial methods. The simulation provides an adequate approximation to this distribution, shown below. (There are 128 distinct values in the simulated permutation distribution.)

hist(a.prm, prob=T, col="skyblue2", 
     main="Simulated Permutation Dist'n")
abline(v = c(-1,1)*mean(d), col="red")

enter image description here

Note: For small amounts of data, I have found permutation tests to be more satisfactory than bootstrap resampling.

In a certain sense, some of the classical nonparametric tests can be considered as 'frozen' permutation tests. (The rank-based metrics are not subject to change.) For our data a one-sample Wilcoxon test, as implemented in R, gives about the same P-value as the permutation test. (I would not want to use a one-sample Wilcoxon test with fewer than about seven differences.)

[1] 0.03125

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.