Return to Answer

With a bootstrap test, we're no longer required to be able to relabel across samples -- we can resample within the samples we have and still get a suitable CI for the difference in means. With some of the usual procedures to improve the properties of the bootstrap, such a test will work very well at these sample sizes.

With a bootstrap test, we're no longer required to be able to relabel across samples -- we can resample within the samples we have and still get a suitable CI for the difference in means. With some of the usual procedures to improve the properties of the bootstrap, such a test might work very well at these sample sizes.

In fact, I think any approach to this problem will be approximate in one way or another; this is the famous Behrens-Fisher problem. As it says at the upper right in the link there, only approximate solutions are known.

For example, imagine we had 334 observations where there was a 90% chance of having an $A$ label, and coming from a normal distribution with $\sigma=1$ and a 10% chance of having a $B$ label and coming from a normal distribution with $\sigma=3$. Further imagine that $\mu_A=\mu_B$. The observations are not exchangeable - in spite of most observations coming from sample $A$, the largest and smallest few observations are much more likely to have come from sample B than sample A and the middle observations are much more likely to have come from sample A (far more than the 90% chance they should have in the observations were exchangeable). This issue affects the distribution of p-values under the null.

What about a bootstrap test

With a bootstrap test, we're no longer required to be able to relabel across samples -- we can resample within the samples we have and still get a suitable CI for the difference in means. This test will work very well.

For example, imagine we had 334 observations where there was a 90% chance of having an $A$ label, and coming from a normal distribution with $\sigma=1$ and a 10% chance of having a $B$ label and coming from a normal distribution with $\sigma=3$. Further imagine that $\mu_A=\mu_B$. The observations are not exchangeable - in spite of most observations coming from sample $A$, the largest and smallest few observations are much more likely to have come from sample B than sample A and the middle observations are much more likely to have come from sample A (far more than the 90% chance they should have in the observations were exchangeable). This issue affects the distribution of p-values under the null. (However, if the sample sizes are equal, the effect is quite small.)

Discussion of the Behrens Fisher problem in Good

The Good Book mentioned by AdamO does discuss this problem on p54-57.

He refers to a result of Romano that states that the permutation test is asymptotically exact providing they have equal sample sizes. Here, of course, they don't - rather than 50-50 they're roughly 90-10.

And when I simulate the equal sample size case (I tried n1=n2=34) the p-value distribution wasn't far off uniform** -- it was off a small amount but not enough to worry about. This is pretty well known and borne out by a number of published simulation studies.

**(I haven't included the code, but it's trivial to adapt the above code to do it - just change n1 to be 34)

Good says the behavior in the equal sample size case works down to quite small sample sizes. I believe him!

What about a bootstrap test?

With a bootstrap test, we're no longer required to be able to relabel across samples -- we can resample within the samples we have and still get a suitable CI for the difference in means. With some of the usual procedures to improve the properties of the bootstrap, such a test will work very well at these sample sizes.