Bootstrap significance test

Question

I'm using the bootstrap method to test my experiment results for significance. I have two sets (say A & B) of 50 grades, for which I want to test whether their means are significantly different. The method I often found while searching for similar problems, is defining a statistic, e.g., mean(A)-mean(B) and checking whether the bootstrapped sampling distribution overlaps zero.

However, I read about another method as well. Here, bootstrapping is applied to the combined grade sets A & B, thus a bootstrap sample is randomly picked from all grades combined. After bootstrapping, checking whether the means of A and B are within the tails of the combined bootstrap distribution would determine whether sets A and B are significantly different. The argument was that, if A and B would not be significantly different, the bootstrap method would show that they both belong to the same distribution.

The second method sounds correct to me, can someone confirm or refute this method?

Answer 1

You're testing the null that the means of both distributions are the same. You're bootstrapping should follow that same null. So you should sample two groups, $\hat{A}$ and $\hat{B}$ where each member of both $\hat A$ and $\hat B$ is drawn from the combined $A$ and $B$. This represents the null that both come from a single population. Then form the statistic: $$\text{mean}\left(\hat A\right) - \text{mean}\left(\hat B\right).$$ Do this a large number of times, and come up with a bootstrapped distribution for this statistic. Then see whether your observed $\text{mean}( A) - \text{mean}( B)$ falls suitably far out in the tails of the bootstrapped distribution for your desired level of significance.

I should point out that for a "suitably large" sample (conventionally interpreted to be $>30$ observations), the $t$-test will work fine because the CLT will apply, and the means of both samples will be normally distributed. So bootstrapping is not usually necessary for this test.