# Two-sample test with bootstrap and different sizes

I have two independent i.i.d., respectively $$Y_1,\dots,Y_{n_Y} \sim F_Y$$ of size $$n_Y$$ and $$X_1,\dots,X_{n_X} \sim G_X$$ of size $$n_X$$. I would like to test: $$H_0:\mu_X = \mu_Y$$ versus any alternative hypothesis like: $$H_1:\mu_X \gtreqless \mu_Y$$ with bootstrap. My question is if the following two alternatives for calculating the bootstrap p-values are the same:

• bootstrap $$Y$$ generating independent bootstrap i.i.d. samples $$Y^*$$ of size $$n_Y$$ then bootstrap $$X$$ generating independent bootstrap i.i.d. samples $$X^*$$ of size $$n_X$$ and calculate the test statistic;
• concatenate $$X$$ and $$Y$$ ($$XY = \begin{bmatrix} X\\ Y \end{bmatrix}$$) and bootstrap $$XY$$ generating independent bootstrap i.i.d. samples $$XY^*$$ of size $$n_Y+n_X$$ and calculate the test statistic;

I don't understand why and if the second option makes sense.

In this code from the book Bootstrap Methods With Applications in R by Gerhard Dikta and Marsel Scheer (2021) I think they consider the second option:

twoSampleLocModBootpValue <-function(x, y, alternative = c("two-sided", "less", "greater"), R = 999){

# x -observed data (first sample)
# y -observed data (second sample)
# alternative -specifies the alternative hypothesis
# R -number of MC simulations

alternative <-match.arg(alternative)

n1 <-length(x)
n2 <-length(y)

# test statistic
tstat <-function(d, i){
boot.xy <-d[i]

x <-boot.xy[1:n1]
y <-boot.xy[-(1:n1)]
s <-sqrt( ((n1-1) * var(x) + (n2 -1) * var(y)) / (n1 + n2 -2))
mu.x <-mean(x)
mu.y <-mean(y)
(mu.x -mu.y) / s / sqrt(1 / n1 + 1 / n2)
}

xy <-c(x,y)

# test statistics for the observed data
t0 <-tstat(xy, 1:(n1 + n2))

#Rresampledteststatistics
bt<-boot::boot(xy,tstat,R=R)\$t[,1]

#return p-value
if(alternative=="greater") return(c(pvalue=mean(bt>t0)))
if(alternative=="less") return(c(pvalue=mean(bt<t0)))
c(pvalue=mean(abs(bt)>abs(t0)))
}



Then applying the function to datasets such that the null hypothesis is true:

#H0c orrect, mu_x equals mu_y
set.seed(123,kind="Mersenne-Twister",normal.kind="Inversion")
twoSampleLocModBootpValue(rnorm(10,mean=3,sd=2), rnorm(20,mean=3,sd=2), alternative="greater")

##pvalue
##0.3143143


Either of those approaches could be appropriate, depending on the initial sampling scheme. Standard bootstrapping from a data sample should represent the same type of sampling as was done to obtain the original sample from the full population.

So, if you had two separate populations $$X$$ and $$Y$$ and deliberately chose respectively $$n_X$$ and $$n_Y$$ separately from each of those populations, then the first approach is appropriate.

Instead, say that you sampled $$n$$ members of a single population who were distinguished only thereafter by their characteristics $$X$$ and $$Y$$. In that case, you ended up with $$n_X$$ and $$n_Y$$ based on the random sampling. Then the second approach best represents the way you collected the data sample.

• For example, if you obtained experimental data where X was the outcome of the control group and Y of the treatment - in such a case, you would rather want to use method 1 - else outcomes from treated individuals might sometimes also be sampled 'into' the control group and vice versa. Dec 21, 2021 at 16:25

Method 1 determines the sampling distribution of the test statistic under the observed data distributions. I'm not sure this is particularly useful. I suppose it can give you a 95% confidence interval for the test statistic. I would use Method 1 to construct a 95% confidence interval for the difference in means rather than perform a hypothesis test.

Often for a hypothesis test you want to determine the sampling distribution of the test statistic under then NULL hypothesis. Then comparing the observed test statistic to the bootstrapped null distribution gives you a p-value.

Method 2 potentially gives you the desired sampling distribution. Randomly sample with replacement nX observations from the combined set (XY) to be your bootstrap resample from X and sample with replacement nY observations from the combined set (XY) to be your bootstrap resample from Y. Compute your test statistic. Note, muX = muY for these resamples. Repeat 1000 times and compare your observed test statistic to the 1000 simulated test statistics.