Bootstrap two-sample t test

Question

I'd like to bootstrap a two sample t-test. My DV is some psychological variable. I have two groups (women and men), unequal sizes and I do not assume equal variances. I'm not sure if my code or/and my thinking is correct, 'cause in the end I got 0 t-statistics greater than t-statistic from original data.

group_k  # women: N=377
group_m  # men:   N=306
t.est <- t.test(group_k, group_m, var.equal=FALSE)$stat
#        t 
# 5.659757

nullA <- group_k - mean(group_k, na.rm=T)
nullB <- group_m - mean(group_m, na.rm=T)
set.seed(1)
b <- function(){
  A <- sample(nullA, 200, replace=T)  # is 200-element from 377-element sample ok? 
  B <- sample(nullB, 200, replace=T) 
  stud_test <- t.test(A, B, var.equal=FALSE)
  stud_test$stat
}
t.stat.vect = vector(length=10000)
t.vect <- replicate(10000, b())

1 - mean(t.est>t.vect)
# [1] 0 :(

I have some additional questions:

1. Why not bootstrapping simply differences between women and men?
2. How to choose bootstrap sample size? In other words, is 200-elements from 377- and 306-element groups OK? Should they be 377 and 306, respectively, as this post recommends?

The idea behind subtracting means was here - gung's reply. I thought that it can be directly taken from ANOVA case to Student's t test.

[UPDATE 13XII] I corrected my code, but results are still awkward to me:

t.est <- t.test(group_k, group_m, var.equal=FALSE)$stat
# t = 5.6598, df = 255.185, p-value = 4.066e-08

b <- function(){
  A <- sample(group_k, 377, replace=T)  
  B <- sample(group_m, 306, replace=T) 
  stud_test <- t.test(A, B, var.equal=FALSE)
  stud_test$stat
}
t.stat.vect = vector(length=10000)
t.vect <- replicate(10000, b())

1 - mean(t.est>t.vect)
[1] 0.5042

Is it possible that using original samples the difference between means is "so significant" (p-value = 4.066e-08), but the bootstrap samples shows that actually it's not (0.5042) ??

Answer 1

As @Tim notes, your bootsamples should have the same $n_j$s as your original data.

Next, recognize that there are several ways to bootstrap: e.g., you can bootstrap your data directly or bootstrap a test statistic, you can bootstrap your sampling distribution or a null distribution, etc. You need to make sure you understand which kind of thing you're doing. You can bootstrap simply the mean difference, if you want to. In the linked post, I bootstrapped the null distribution of the test statistic. That is essentially what you are doing in your code.

Also, because of the ways tests can differ, the bootstrapping strategy may need to be customized to the test you want to perform. In the linked post, I bootstrapped an $F$-statistic, but the way the $F$-test works is somewhat different from how a $t$-test works. Since you are bootstrapping the test statistic, you are somewhat safe from that.

In your case, think about the logic of the type of bootstrap you used. You bootstrapped a null sampling distribution for your $t$-statistic. Your observed $t$-statistic is so extreme that none of the bootstrapped $t$s overlapped with it. The implication of that is that the probability ($p$-value) of getting a $t$-statistic as far or further from $0$ from your bootstrapped null sampling distribution is $< (1/10000) / 2$. In other words, your result is highly significant. (However, you should re-do your bootstrap using the correct $n_j$s before you go with this result.)

Answer 2

First of all, with bootstrap you sample $N$ cases out of $N$ cases in your data. So the number of observations to choose is simple.

And, yes, you can do:

f <- function() {
   A <- sample(group_A, 377, replace=T) 
   B <- sample(group_B, 377, replace=T)
   mean(A)-mean(B)
}
replicate(1000, f())

but this is a different approach than using t-test, because it will provide you information of the possible range of differences between those two means. You could use this range in similar fashion as you could use boxplots for (informal) analysis of differences between the means. For hypothesis testing it is however better to use a classical bootstrap that computes t-test on every iteration and outputs t-statistics. The reason for that is that t-test does not only compute the difference between means, but also takes into consideration variances of the two groups.

For gaining deeper understanding of bootstrap I would recommend you the classic 1979 paper by Efron and his very readable book.

Answer 3

One more thing: if you're not assuming equal variance, you might want to consider Welch's t-test (http://beheco.oxfordjournals.org/content/17/4/688.full)

Answer 4

I would condition on the total sample size, not fix the group sizes. Of course, if your group sizes are fixed in advance then do condition on those, my answer is for the case where they are not.

I construct a two column matrix with the first column being an indication of group membership and the second column the observed values for the two groups stacked underneath each other. Then I bootstrap rows of the matrix and calculate the observed difference in means between the groups. Lastly I use the bootstrapped differences to calculate an approximate p-value for the test of zero difference in means.

pvalfunc <- function(sims,target=0) { 2*min(mean(sims<target),mean(sims>target)) }

boot.2sdif.test <- function(s1,s2, nboot=9999) {
  n1 <- length(s1); n2 <- length(s2); n <- n1+n2
  X <- cbind(rep(c(1,2), c(n1,n2)), c(s1,s2))
  d <- rep(0,nboot)
  for (i in 1:nboot) {
    b <- X[sample.int(n, n, T),]
    d[i] <- mean(b[b[,1]==1, 2]) - mean(b[b[,1]==2, 2])
  }
  return(pvalfunc(d))

(pvalue <- boot.2sdif.test(group_k, group_m))