Bootstrap two-sample t test

I'd like to bootstrap a two sample t-test. My DV is some psychological variable. I have two groups (women and men), unequal sizes and I do not assume equal variances. I'm not sure if my code or/and my thinking is correct, 'cause in the end I got 0 t-statistics greater than t-statistic from original data.

group_k  # women: N=377
group_m  # men:   N=306
t.est <- t.test(group_k, group_m, var.equal=FALSE)$stat # t # 5.659757 nullA <- group_k - mean(group_k, na.rm=T) nullB <- group_m - mean(group_m, na.rm=T) set.seed(1) b <- function(){ A <- sample(nullA, 200, replace=T) # is 200-element from 377-element sample ok? B <- sample(nullB, 200, replace=T) stud_test <- t.test(A, B, var.equal=FALSE) stud_test$stat
}
t.stat.vect = vector(length=10000)
t.vect <- replicate(10000, b())

1 - mean(t.est>t.vect)
# [1] 0 :(


1. Why not bootstrapping simply differences between women and men?
2. How to choose bootstrap sample size? In other words, is 200-elements from 377- and 306-element groups OK? Should they be 377 and 306, respectively, as this post recommends?

The idea behind subtracting means was here - gung's reply. I thought that it can be directly taken from ANOVA case to Student's t test.

[UPDATE 13XII] I corrected my code, but results are still awkward to me:

t.est <- t.test(group_k, group_m, var.equal=FALSE)$stat # t = 5.6598, df = 255.185, p-value = 4.066e-08 b <- function(){ A <- sample(group_k, 377, replace=T) B <- sample(group_m, 306, replace=T) stud_test <- t.test(A, B, var.equal=FALSE) stud_test$stat
}
t.stat.vect = vector(length=10000)
t.vect <- replicate(10000, b())

1 - mean(t.est>t.vect)
[1] 0.5042


Is it possible that using original samples the difference between means is "so significant" (p-value = 4.066e-08), but the bootstrap samples shows that actually it's not (0.5042) ??

• You get 0 because all the bootstrapped t-statistics (t.vect) are less than t.est. mean(c(TRUE, TRUE, TRUE, ..., TRUE)) is 1. So, 1-1 = 0. – cdeterman Dec 11 '14 at 17:11

As @Tim notes, your bootsamples should have the same $n_j$s as your original data.

Next, recognize that there are several ways to bootstrap: e.g., you can bootstrap your data directly or bootstrap a test statistic, you can bootstrap your sampling distribution or a null distribution, etc. You need to make sure you understand which kind of thing you're doing. You can bootstrap simply the mean difference, if you want to. In the linked post, I bootstrapped the null distribution of the test statistic. That is essentially what you are doing in your code.

Also, because of the ways tests can differ, the bootstrapping strategy may need to be customized to the test you want to perform. In the linked post, I bootstrapped an $F$-statistic, but the way the $F$-test works is somewhat different from how a $t$-test works. Since you are bootstrapping the test statistic, you are somewhat safe from that.

In your case, think about the logic of the type of bootstrap you used. You bootstrapped a null sampling distribution for your $t$-statistic. Your observed $t$-statistic is so extreme that none of the bootstrapped $t$s overlapped with it. The implication of that is that the probability ($p$-value) of getting a $t$-statistic as far or further from $0$ from your bootstrapped null sampling distribution is $< (1/10000) / 2$. In other words, your result is highly significant. (However, you should re-do your bootstrap using the correct $n_j$s before you go with this result.)

• should it be: mean(t.est>t.vect) instead of 1 - mean(t.est>t.vect)? I copied this formula after your post with F-statistics, but now I'm not sure if it's ok... – Lil'Lobster Dec 13 '14 at 19:02
• @Lili, you are trying to get the % of the bootstrapped sampling distribution that lies further away from 0 than your estimated t statistic. You can use 1 - mean(t.est>t.vect), or you could use mean(t.vect>t.est) (which will give you the same answer). – gung Dec 13 '14 at 22:45

First of all, with bootstrap you sample $N$ cases out of $N$ cases in your data. So the number of observations to choose is simple.

And, yes, you can do:

f <- function() {
A <- sample(group_A, 377, replace=T)
B <- sample(group_B, 377, replace=T)
mean(A)-mean(B)
}
replicate(1000, f())


but this is a different approach than using t-test, because it will provide you information of the possible range of differences between those two means. You could use this range in similar fashion as you could use boxplots for (informal) analysis of differences between the means. For hypothesis testing it is however better to use a classical bootstrap that computes t-test on every iteration and outputs t-statistics. The reason for that is that t-test does not only compute the difference between means, but also takes into consideration variances of the two groups.

For gaining deeper understanding of bootstrap I would recommend you the classic 1979 paper by Efron and his very readable book.

• I updated my question with the answer about subtracting means. – Lil'Lobster Dec 11 '14 at 17:26
• sorry if this question is inappropriate, but how to get this book being in Poland? I ask you, 'cause it's written you're from Poland, and I assume you know the reality of an access to knowledge here. – Lil'Lobster Dec 12 '14 at 12:34
• @gung OK, sorry my mistake, I misread that subtracting the mean was inside for loop, so it did not make sens for me. My mistake. – Tim Dec 12 '14 at 17:00
• @Tim, no problem. – gung Dec 12 '14 at 17:01
• I think you can use the sampling distribution of the unstandardized mean difference inferentially. – gung Dec 12 '14 at 17:08

One more thing: if you're not assuming equal variance, you might want to consider Welch's t-test (http://beheco.oxfordjournals.org/content/17/4/688.full)

• I don't know if you are familiar with R, but in t.test the var.equal=FALSE parameter means that Welch correction is used. – Tim Dec 12 '14 at 11:31
• @Tim thank you for your help :) I work in R, also I know Welch correction, but I do want to bootstrap and I followed steps by gung, actually not fully understanding why to subtract those means. – Lil'Lobster Dec 12 '14 at 12:04
• @Lili generally you could think of bootstrap as sampling from the data in similar fashion as you sampled your data from the population. In many cases it is pretty simple procedure. – Tim Dec 12 '14 at 12:16