Bootstrap two-sample t test

I'd like to bootstrap a two sample t-test. My DV is some psychological variable. I have two groups (women and men), unequal sizes and I do not assume equal variances. I'm not sure if my code or/and my thinking is correct, 'cause in the end I got 0 t-statistics greater than t-statistic from original data.

group_k  # women: N=377
group_m  # men:   N=306
t.est <- t.test(group_k, group_m, var.equal=FALSE)$stat # t # 5.659757 nullA <- group_k - mean(group_k, na.rm=T) nullB <- group_m - mean(group_m, na.rm=T) set.seed(1) b <- function(){ A <- sample(nullA, 200, replace=T) # is 200-element from 377-element sample ok? B <- sample(nullB, 200, replace=T) stud_test <- t.test(A, B, var.equal=FALSE) stud_test$stat
}
t.stat.vect = vector(length=10000)
t.vect <- replicate(10000, b())

1 - mean(t.est>t.vect)
# [1] 0 :(


1. Why not bootstrapping simply differences between women and men?
2. How to choose bootstrap sample size? In other words, is 200-elements from 377- and 306-element groups OK? Should they be 377 and 306, respectively, as this post recommends?

The idea behind subtracting means was here - gung's reply. I thought that it can be directly taken from ANOVA case to Student's t test.

[UPDATE 13XII] I corrected my code, but results are still awkward to me:

t.est <- t.test(group_k, group_m, var.equal=FALSE)$stat # t = 5.6598, df = 255.185, p-value = 4.066e-08 b <- function(){ A <- sample(group_k, 377, replace=T) B <- sample(group_m, 306, replace=T) stud_test <- t.test(A, B, var.equal=FALSE) stud_test$stat
}
t.stat.vect = vector(length=10000)
t.vect <- replicate(10000, b())

1 - mean(t.est>t.vect)
[1] 0.5042


Is it possible that using original samples the difference between means is "so significant" (p-value = 4.066e-08), but the bootstrap samples shows that actually it's not (0.5042) ??

• You get 0 because all the bootstrapped t-statistics (t.vect) are less than t.est. mean(c(TRUE, TRUE, TRUE, ..., TRUE)) is 1. So, 1-1 = 0. Commented Dec 11, 2014 at 17:11

As @Tim notes, your bootsamples should have the same $n_j$s as your original data.

Next, recognize that there are several ways to bootstrap: e.g., you can bootstrap your data directly or bootstrap a test statistic, you can bootstrap your sampling distribution or a null distribution, etc. You need to make sure you understand which kind of thing you're doing. You can bootstrap simply the mean difference, if you want to. In the linked post, I bootstrapped the null distribution of the test statistic. That is essentially what you are doing in your code.

Also, because of the ways tests can differ, the bootstrapping strategy may need to be customized to the test you want to perform. In the linked post, I bootstrapped an $F$-statistic, but the way the $F$-test works is somewhat different from how a $t$-test works. Since you are bootstrapping the test statistic, you are somewhat safe from that.

In your case, think about the logic of the type of bootstrap you used. You bootstrapped a null sampling distribution for your $t$-statistic. Your observed $t$-statistic is so extreme that none of the bootstrapped $t$s overlapped with it. The implication of that is that the probability ($p$-value) of getting a $t$-statistic as far or further from $0$ from your bootstrapped null sampling distribution is $< (1/10000) / 2$. In other words, your result is highly significant. (However, you should re-do your bootstrap using the correct $n_j$s before you go with this result.)

• should it be: mean(t.est>t.vect) instead of 1 - mean(t.est>t.vect)? I copied this formula after your post with F-statistics, but now I'm not sure if it's ok... Commented Dec 13, 2014 at 19:02
• @Lili, you are trying to get the % of the bootstrapped sampling distribution that lies further away from 0 than your estimated t statistic. You can use 1 - mean(t.est>t.vect), or you could use mean(t.vect>t.est) (which will give you the same answer). Commented Dec 13, 2014 at 22:45

First of all, with bootstrap you sample $N$ cases out of $N$ cases in your data. So the number of observations to choose is simple.

And, yes, you can do:

f <- function() {
A <- sample(group_A, 377, replace=T)
B <- sample(group_B, 377, replace=T)
mean(A)-mean(B)
}
replicate(1000, f())


but this is a different approach than using t-test, because it will provide you information of the possible range of differences between those two means. You could use this range in similar fashion as you could use boxplots for (informal) analysis of differences between the means. For hypothesis testing it is however better to use a classical bootstrap that computes t-test on every iteration and outputs t-statistics. The reason for that is that t-test does not only compute the difference between means, but also takes into consideration variances of the two groups.

For gaining deeper understanding of bootstrap I would recommend you the classic 1979 paper by Efron and his very readable book.

• I updated my question with the answer about subtracting means. Commented Dec 11, 2014 at 17:26
• sorry if this question is inappropriate, but how to get this book being in Poland? I ask you, 'cause it's written you're from Poland, and I assume you know the reality of an access to knowledge here. Commented Dec 12, 2014 at 12:34
• @gung OK, sorry my mistake, I misread that subtracting the mean was inside for loop, so it did not make sens for me. My mistake.
– Tim
Commented Dec 12, 2014 at 17:00
• @Tim, no problem. Commented Dec 12, 2014 at 17:01
• I think you can use the sampling distribution of the unstandardized mean difference inferentially. Commented Dec 12, 2014 at 17:08

One more thing: if you're not assuming equal variance, you might want to consider Welch's t-test (http://beheco.oxfordjournals.org/content/17/4/688.full)

• I don't know if you are familiar with R, but in t.test the var.equal=FALSE parameter means that Welch correction is used.
– Tim
Commented Dec 12, 2014 at 11:31
• @Tim thank you for your help :) I work in R, also I know Welch correction, but I do want to bootstrap and I followed steps by gung, actually not fully understanding why to subtract those means. Commented Dec 12, 2014 at 12:04
• @Lili generally you could think of bootstrap as sampling from the data in similar fashion as you sampled your data from the population. In many cases it is pretty simple procedure.
– Tim
Commented Dec 12, 2014 at 12:16

I would condition on the total sample size, not fix the group sizes. Of course, if your group sizes are fixed in advance then do condition on those, my answer is for the case where they are not.

I construct a two column matrix with the first column being an indication of group membership and the second column the observed values for the two groups stacked underneath each other. Then I bootstrap rows of the matrix and calculate the observed difference in means between the groups. Lastly I use the bootstrapped differences to calculate an approximate p-value for the test of zero difference in means.

pvalfunc <- function(sims,target=0) { 2*min(mean(sims<target),mean(sims>target)) }

boot.2sdif.test <- function(s1,s2, nboot=9999) {
n1 <- length(s1); n2 <- length(s2); n <- n1+n2
X <- cbind(rep(c(1,2), c(n1,n2)), c(s1,s2))
d <- rep(0,nboot)
for (i in 1:nboot) {
b <- X[sample.int(n, n, T),]
d[i] <- mean(b[b[,1]==1, 2]) - mean(b[b[,1]==2, 2])
}
return(pvalfunc(d))

(pvalue <- boot.2sdif.test(group_k, group_m))