I have a data set with two classes and completely unbalanced samples sizes between the two classes. Note that even though I am showing some R code here, this is not a programming question.

  class   value
1     1  0.0469
2     1 -1.6735
3     1  0.7994
4     1  0.2140
5     1 -0.7963
6     1 -1.2144

Class sizes are as follows:


  0   1 
  5 456 

T.test says there is a difference in mean with essentially zero p.value:

t.test(value ~ class, data = df)

    Welch Two Sample t-test

data:  value by class
t = 9.391, df = 46.315, p-value = 2.7e-12
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 0.2868030 0.4432572
sample estimates:
mean in group 0 mean in group 1 
      0.7482200       0.3831899 

A simulation shows that the p.value is much, much higher:

results <- vector(length = 100000)
for (i in 1:100000) {
    x <- sample(df$value, sum(df$class == 1))
    y <- sample(df$value,  sum(df$class == 0))
    results[i] <- abs(mean(x) - mean(y))

P.value provided by this method is:

mean(results > 0.3650301)
[1] 0.23134

I am tempted to discard the t.test results and run the permutation based method on the rest of my data and rely on that method.

What makes these two to be so divergent? How does one go about resolving this type of situation?

  • $\begingroup$ Am I correctly interfering that you have only 5 examples of class 0, but 456 of class 1? $\endgroup$ Apr 4, 2016 at 19:13
  • $\begingroup$ That is correct. $\endgroup$ Apr 4, 2016 at 19:23
  • $\begingroup$ Your simulation doesn't appear to have much to do with the t-test. If you wish to compare the numbers in results, then they had better be t-statistics. But they're not: they are merely absolute values of differences in means. It's meaningless to compare their mean to 0.3650301 (which appears to be the Welch t statistic of your data). $\endgroup$
    – whuber
    Apr 4, 2016 at 20:32
  • 1
    $\begingroup$ I am computing how many times the permuted draws result in a mean difference as extreme as the difference in means of the samples. It says in about 23% of the cases. If I computed a t-statistic (by dividing with pooled standard deviation of the two samples), do I then compute how many times it's absolute value is greater than 2? $\endgroup$ Apr 4, 2016 at 20:39
  • 1
    $\begingroup$ Why would this statistic be expected to have a t-distribution here? $\endgroup$
    – Glen_b
    Apr 5, 2016 at 0:35


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.