I am dealing with data with unbalanced sample sizes (12k vs 18k) and variance (0.2 vs 0.4) and want to perform a two sample t-test. With the unbalanced features, I am thinking about applying a bootstrapped t-test. Can I just re-sample from the larger data-set and perform t-test many times then get the average?

To be more specific about the data I am dealing with:
Set 1: 12K (var 0.02) vs 18K (var 0.04),
Set 2: 700 (0.02) vs 900 (0.04).

I also need to compare 12K from set1 to 700 from set 2. It seems that 1,2 are OK for the Welch two sample t-test, but I am more concerned about the third one.

t.test.btsp <- function(data1, data2, n=1000, n1, n2)
  t.values = numeric(n)
  for(i in 1:n)
    group1      <- sample(data1$v1, size=n1, replace = T)
    group2      <- sample(data2$v1, size=n2, replace = T)
    t.values[i] <- t.test(group1, group2)$statistic
a <- t.test.btsp(data1, data2, n=1000, n1=700, n2=700)
quantile(a, c(0.05,0.95))

Is this the right way to get the CI?

  • 3
    $\begingroup$ Many t-test applications are overkill. They are needed only when the difference of means is (a) of practical importance and (b) of the same order of magnitude as the standard errors of the means. Here, your standard errors are around $0.002$ and $0.003$, so if the difference of means is much larger than $0.01$, there's no question it is significant and it won't matter what test you apply. In particular, there would be no point in applying any procedure that takes more than a few seconds of your time and the computer's time total. $\endgroup$ – whuber May 28 '14 at 22:12

You can perform the t-test many times, but the resampling should be across observations that are exchangeable.

So you shouldn't resample across groups for several reasons. You also shouldn't average the results.

You could resample within groups and then form a bootstrap CI for the difference in means.

But you may not need so much effort.

You might consider a Welch t-test for example.

In fact with those sample sizes, unless the data are very strongly non-normal (extreme heavy-tails, say), or the variation is gigantic, you should be able to treat the sample standard deviations as population sds and do an unequal-variance z-test.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.