Bootstrap for hypothesis testing - beginners question

Question

Scenario: I have two measurement tools A and B and only about n=5-10 measurements of the same object for each tool.

I want to test if there is a difference in the mean of the measurements between the two tools. I want to use the bootstrap hypothesis approach creating 1000 bootstrapped samples with replacement for each tool (or do I need to take the values of both tools together and create bootstrap samples?). Here I get 1000 estimated means and a confidence interval from the bootstrapped sample distribution.

Two separate questions:

1.) Is it possible to use these 1000 means for each tool as "values" in a non-parametric test (for example a Mann-Whitney-U-test)? I was wondering if this "upsampling" of measurement points is valid?

2.) Is it possible to use the estimated mean and CI of the bootstrap sampling distribution and if the confidence intervals of both groups don't overlap, can I conclude that they are significantly different?

Thank you.

Answer 1

Suppose you have 7 subjects (or objects) and have measures A and B on each, with (fake) data as follows:

Subject      1      2      3      4      5      6      7
A        77.04 111.79 109.82  90.02  97.93  84.01 105.72
B        82.80 117.28 109.58  96.13 100.72  88.07 111.46
D = B-A   5.76   5.49  -0.24   6.11   2.79   4.06   5.74

A paired t test (one-sample t test on difference) shows a significant difference at the 0.3% level, as follows:

d = c(5.76, 5.49, -0.24, 6.11, 2.79, 4.06, 5.74)
t.test(d)$p.val
[1] 0.002778031

But suppose you doubt the normality of the data and worry about using a t test for such a small sample.

A nonparametric permutation test is based on a large number of random permutations of the sign of the differences. For each permutation the mean difference is found. (One says that the difference is the 'metric' for the permutation test.) The observed difference is $\bar D_{obs} = 4.244.$

mean(d)
[1] 4.244286

The P-value of the two-sided permutation test is the proportion of permuted differences that equals or exceeds the observed difference in absolute value. In this case, the P-value is 0.03.

set.seed(504)
a.prm = replicate(10^5, mean(sample(c(-1,1),7,rep=T)*d))
mean(abs(a.prm) >= abs(mean(d)))
[1] 0.03029

With some trouble one could obtain the exact permutation distribution by combinatorial methods. The simulation provides an adequate approximation to this distribution, shown below. (There are 128 distinct values in the simulated permutation distribution.)

hist(a.prm, prob=T, col="skyblue2", 
     main="Simulated Permutation Dist'n")
rug(a.prm)
abline(v = c(-1,1)*mean(d), col="red")

Note: For small amounts of data, I have found permutation tests to be more satisfactory than bootstrap resampling.

In a certain sense, some of the classical nonparametric tests can be considered as 'frozen' permutation tests. (The rank-based metrics are not subject to change.) For our data a one-sample Wilcoxon test, as implemented in R, gives about the same P-value as the permutation test. (I would not want to use a one-sample Wilcoxon test with fewer than about seven differences.)

wilcox.test(d)$p.val
[1] 0.03125