What to bootstrap for hypothesis testing

Question

I have a small question about the concept behind hypothesis testing using bootstrap. Assume that I need to evaluate two independent population mean differences: population a and population b. My doubt is the following:

1. Should I apply bootstrap on a single population, and check the difference of the mean after that?

   Mean[BOOT(a)-BOOT(b)]
   

2. Alternatively, should I compute di difference: Mean(a)-Mean(b) and then apply bootstrap?

   BOOT[Mean(a)-Mean(b)]
   

I used this code by using the second approach:

set.seed(123)
a <- rnorm(100)
b <- rnorm(100)
hist(a)
hist(b)

c = a-b
hist(c)

boot_1 = function(R,dati_oss){
  n = length(dati_oss)
  media_boot = vector("numeric",R)
  for(i in 1:R){
    ind = sample(1:n,replace=T)
    media_boot[i] = mean(dati_oss[ind])
  }
  return(media_boot)
}

res=boot_1(500000,c)

hist(res)

stat = matrix(c(mean(c), mean(res), mean(res)-mean(c), sqrt(var(res)),
              as.vector(quantile(res, c(0.025,0.975)))), 1, 6)
colnames(stat) = c("Observed", "Mean-boot", "Bias", "SE", "0.95LCI", "0.95UCI")
row.names(stat) = c("Mean")
stat

Answer 1

The basic principle to apply, quoting @MichaelChernick, is: "Sampling with replacement behaves on the original sample the way the original sample behaves on a population."

Think about how you analyzed the original sample. You took the sample, calculated the mean of each of the 2 groups, and determined the difference between their mean values to get an estimate of the a-b difference.

So you proceed similarly with each bootstrapped resample: resample from the original sample, calculate the mean of each group, and determine the difference between the means of the two groups as represented in the resample. Do this a large number of times to estimate the distribution of a-b differences. Compare the mean of the bootstrapped a-b differences against the a-b difference found in the original sample to estimate the bias in the original a-b difference.

Note that the way you design the resampling might depend on the original study design. If you had two independent populations from which you took samples, then the resampling should proceed comparably, within each of the populations. If you sampled from a mixed population in which individual cases were labeled as belonging to population a versus b, then you should resample from a pool of all the cases in the original sample.

Answer 2

Without going into the coding of it, consider what happens if you calculate Mean(a)-Mean(b) first (proposal 2). You calculate Mean(a) and get a single number, then you calculate Mean(b) and get a single number. Then you take your two numbers, and calculate Mean(a)-Mean(b) to get a single value for your difference. No matter how many times you sample/bootstrap this single value, you will get the same number. Try it out!

Whereas if you take multiple samples of your population A and population B, then calculate the difference of means of your samples (proposal 1) you will get slightly different combinations nearly every time you sample (assuming you don't re-set the random seed at the wrong point in your code), so you will get a range of values for the difference.

I would consider proposal 1 to be a form of bootstrapping, but I wouldn't say the same about proposal 2!