Bootstrapping a t statistic - Rationale and procedure

Question

I was hoping someone could help me out with this. I've seen similar questions on the forum, but I need to know if I've understood the correct rationale and procedure for bootstrapping for my particular need.

In my study, I have two independent groups of equal sample size (22 individuals in each group). I am interested in the difference between these two groups, so I've run a simple t-test. The resulting t-value was 1.39, with a p value of .17. The assumptions of the test look good; Equal variance between groups, and variable is close to normally-distributed (but sample sizes are small).

My supervisor advised me to run "simulations" on the test statistic, since the p value was relatively low (to avoid a type-II error). After reading up on resampling techniques, I've concluded that it is feasible to bootstrap the test-statistic, and then explore the range of t values and associated p values to determine the precision of my estimate. Could someone please verify if this is an appropriate rationale for bootstrapping?

As I understand, one procedure for this would be to: a) randomly extract (With replacement) 22 cases for each of my two Groups, b) perform a t-test on these new samples, c) repeat e.g. 1000 times, d) look/plot the ranges of t and p values, and use these to infer if there is a difference between my groups. I am planning on using R for bootstrapping.

Would this be an appropriate way of bootstrapping in my example? I've read that one can bootstrap data directly, or bootstrap just the test statistic. I guess I really want the latter one, but not sure what the difference between these two approaches are.

Answer 1

Your null hypothesis is that there’s no difference in means between groups, so you should choose a randomized test that checks this. Randomly permute the group assignments (labels) and compare the means (e.g. calculate the $t$ statistic). Doing this many times will give you the null distribution of the statistic. Afterward, you could compare how likely is the statistic calculated on real data given the null distribution.

No need to collect the $p$-values on the simulated data, just calculate it as

$$ p_\text{perm} = \frac{1}{R} \sum_{i=1}^R \,\mathbb{I}\{ t_\text{real} \ge t_{\text{perm}_i} \} $$

where $R$ is the number of simulated outcomes. Of course, depending on your hypothesis, choose an appropriate comparison, e.g. $\ge$, $\lt$, two-tailed difference, etc.

Answer 2

There are a lot of ways to bootstrap, and the literature on the subject is quite substantial. This also means that there are many opinions, so you might want to run it by you supervisor before handing anything in.

That said, I think that your rationale is fine. The procedure would be as you describe, resample from each group with replacement (each sample should be of the same size as the original), calculate sample mean and standard error, calculate the t statistic, repeat m times. Plot the distribution and determine the lower and upper bounds, depending on the significance level. If your assumptions are truly valid, I doubt you will learn anything new.