Why would I want to bootstrap when computing an independent sample t-test? (how to justify, interpret, and report a bootstrapped t-test)

Question

Let's say I have two conditions, and my sample size for the two conditions is extremely low. Let's say I only have 14 observations in the first condition and 11 in the other. I want to use the t-test to test if the mean differences are significantly different from one another.

First, I am a little confused about the normality assumption of the t-test, which might be why I am not totally getting bootstrapping. Is the assumption for the t-test that (A) the data is sampled from a normal population, or (B) that your sample distributions have Gaussian properties? If it is (B) then it isn't really an assumption, right? You can just plot a histogram of your data and see if it is normal or not. If my sample size is low though, I won't have enough data points to see if my sample distribution is normal though.

This is where I think bootstrapping comes in. I can bootstrap to see if my sample is normal, right? At first I thought that bootstrapping would always result in a normal distribution, but this isn't the case(Can Bootstrap Resampling be used to Calculate a Confidence Interval for the Variance of a Data Set? statexchange statexchange). So, one reason you would bootstrap is to be more certain of the normality of your sample data, correct?

At this point I become thoroughly confused though. If I perform a t-test in R with the t.test function and I put the bootstrapped sample vectors in as the two independent samples, my t value simply becomes insanely significant. Am I not doing the bootstrapped t-test right? I must not, because all bootstrapping is doing is just making my t value larger, wouldn't this happen in every case? Do people not perform a t-test on the bootstrapped samples?

Lastly, what is the benefit of computing confidence intervals on a bootstrap versus computing confidence intervals on our original sample? What do these confidence intervals tell me that confidence intervals on the original sample data don't?

I guess I am confused on (A) why to use a bootstrap if it will just make my t value more significant, (B) unsure of the correct way to utilize bootstrapping when running an independent sample t-test, and (C) unsure how to report the justification, execution, and results of bootstrapping in independent t-test situations.

Answer 1

There are several misunderstandings in you post (some of which are common and you may have been told the wrong thing because the person telling you was just passing on the misinformation).

First is that bootstrap is not the savior of the small sample size. Bootstrap actually fairs quite poorly for small sample sizes, even when the population is normal. This question, answer, and discussion should shed some light on that. Also the article here gives more details and background.

Both the t-test and the bootstrap are based on sampling distributions, what the distribution of the test statistic is.

The exact t-test is based on theory and the condition that the population/process generating the data is normal. The t-test happens to be fairly robust to the normality assumption (as far as the size of the test goes, power and precision can be another matter) so for some cases the combination of "Normal enough" and "Large sample size" means that the sampling distribution is "close enough" to normal that the t-test is a reasonable choice.

The bootstrap instead of assuming a normal population, uses the sample CDF as an estimate of the population and computes/estimates (usually through simulation) the true sampling distribution (which may be normalish, but does not need to be). If the sample does a reasonable job of representing the population then the bootstrap works well. But for small sample sizes it is very easy for the sample to do a poor job of representing the population and the bootstrap methods do lousy in those cases (see the simulation and paper referenced above).

The advantage of the t-test is that if all the assumptions hold (or are close) then it works well (I think it is actually the uniformly most powerful test). The disadvantage is that it does not work well if the assumptions are not true (and not close to being true) and there are some cases where the assumptions make a bigger differences than in others. And the t-test theory does not apply for some parameters/statistics of interest, e.g. trimmed means, standard deviations, quantiles, etc.

The advantage of the bootstrap is that it can estimate the sampling distribution without many of the assumptions needed by parametric methods. It works for statistics other than the mean and in cases where other assumptions do not hold (e.g. 2 samples, unequal variances). The disadvantage of the bootstrap is that it is very dependent on the sample representing population because it does not have the advantages of other assumptions. The bootstrap does not give you normality, it gives you the sampling distribution (which sometimes looks normal, but still works when it is not) without needing the assumptions about the population.

For t-tests where it is reasonable to assume that the population is normal (or at least normal enough) then the t-test will be best (of the 2).

If you do not have normality and do have small samples, then neither the t-test or the bootstrap should be trusted. For the 2 sample case a permutation test will work well if you are willing to assume equal distributions (including equal variances) under the null hypothesis. This is a very reasonable assumption when doing a randomized experiment, but may not be when comparing 2 separate populations (but then if you believe that 2 populations may have different spreads/shapes then maybe a test of means is not the most interesting question or the best place to start).

With huge sample sizes the large sample theory will benefit both t-tests and bootstrapping and you will see little or no difference when comparing means.

With moderate sample sizes the bootstrap can perform well and may be preferred when you are unwilling to make the assumptions needed for the t-test procedures.

The important thing is to understand the assumptions and conditions that are required for the different procedures that you are considering and to consider how those conditions and deviations from them will affect your analysis and how you believe the population/process that produced your data fits those conditions, simulation can help you understand how the deviations affect the different methods. Remember that all statistical procedures have conditions and assumptions (with the possible exception of SnowsCorrectlySizedButOtherwiseUselessTestOfAnything, but if you use that test then people will make assumptions about you).