In a small sample set, what constitutes unbalanced? I want to do a t-test

Question

I want to do a t-test on my data set. However, my data set is very small and unbalanced (e.g. 10 in one group and 6 in another). Each group has a normal distribution and equal variance according to their individual qq plots and r's var.test. The two groups also have equal variance according to the more conservative levene's test.

1. I know t-tests are more resilient to sample size differences, provided the variance is equal. Is this true for a sample size like mine?

2. I want to compare whether the two groups differ from each other. If 1. is not true, given my sample size, what options do I have? The classic independent two-sample t-test? Welch's t-test? Some non-parametric test?

Answer 1

I believe the main difficulty with unbalanced two-sample t tests, occurs when the two populations have unequal variances and you use the 'pooled' t test. For example, if the smaller sample corresponds to the population with the larger variance, what you believe to be a test at the 5% level, may have an error probability greater than 5%.

However, the Welch two-sample t test does not require equal population variances, and the significance level of a test at the "5% level' truly has a 5% probability of false rejection. Then the only difficulty with imbalance in the two sample sizes may be an inefficient design. For example, if you have resources to use 16 subjects altogether, power (probability of detecting a true difference between population means) will be greater if you use $n_1 = n_2 = 8$ than if you use $n_1 = 5, n_2 = 11.$

set.seed(1234)
x1 = rnorm(5, 100, 10);  x2 = rnorm(11, 115, 4) # unbalanced

t.test(x1, x2)

      Welch Two Sample t-test

data:  x1 and x2
t = -2.753, df = 4.1113, p-value = 0.04969
alternative hypothesis: 
   true difference in means is not equal to 0
95 percent confidence interval:
 -34.53107719  -0.03747153
sample estimates:
mean of x mean of y 
 96.47646 113.76074 

set.seed(1232)
x1 = rnorm(8, 100, 10);  x2 = rnorm(8, 115, 4)  # balanced

t.test(x1, x2)

    Welch Two Sample t-test

data:  x1 and x2
t = -3.2369, df = 9.3901, p-value = 0.009662
alternative hypothesis: 
   true difference in means is not equal to 0
95 percent confidence interval:
 -25.570318  -4.610759
sample estimates:
mean of x mean of y 
 99.60904 114.69958 

Your procedures testing in advance whether the two population variances are equal $\sigma_1^2 = \sigma_2^2$ (with an F-test or Levene test) may seem logical. However, experience has shown that it is better (a) to use the Welch test consistency than (b) to do a 'hybrid' test, checking for unequal variances and branching to pooled or Welch two-sample t, depending on the result of the variance test.

I would not use a nonparametric test (such as the two-sample Wilcoxon test) unless I had serious doubts about the normality of the data. Also, for very small sample sizes, a Wilcoxon test may not be able to give a P-value below 5%.

set.seed(4321)
x1 = rnorm(3, 100, 5);  x2 = rnorm(3, 200, 5)

wilcox.test(x1, x2)

        Wilcoxon rank sum test

data:  x1 and x2
W = 0, p-value = 0.1
alternative hypothesis: 
   true location shift is not equal to 0

t.test(x1, x2)

        Welch Two Sample t-test

data:  x1 and x2
t = -33.796, df = 3.5837, p-value = 1.28e-05
alternative hypothesis: 
   true difference in means is not equal to 0
95 percent confidence interval:
 -112.6878  -94.8295
sample estimates:
mean of x mean of y 
 100.1121  203.8707 

Answer 2

As far as I know, the differences in sample sizes is not a problem in your case. (I would rather worry for the small sample sizes in both groups.)

Here, I would rather go for a Wilcoxon test: it might be difficult to be sure (by any means) that your distribution is gaussian or that your variances are equal with as few individuals as 6. I think you should prefer a non parametric test that does not make any strong requirement about the underlying distributions, since it is something you cannot prove in a reliable way here.