CLT for t-test with unequal sample size; one group < 30?

Question

CrossValidated has many discussions on how unequal variances are not a practical issue for two-sample t-tests when Welch correction is used and on how normality assumptions do not play a role (in Type-1-error control) once we have a large enough sample size (some propose a rule of thumb of $n \geq 30$, some $n \geq 50$). The reasoning is that with growing numbers the Central Limit Theorem will ensure normal distribution of the means to be compared.

In those threads I fail to find definitions of $n$. Do these rules of thumb define $n$ as the number of all observations in the test or the number in the smallest group to be tested?

If I had $n_1 = 500$ to compare to $n_2 = 4$ observations. Would the many data on group 1 "heal" the small sample of group 2?

Additional Question: And if not, could I do a permutation test of means on that kind of data? (That seems to be sometimes advised as a cure all but usually, there is not such a thing as a cure all and I may not understand it's limitations).

Answer 1

One advantage of modern computers (beyond being able to ask questions of people across the world) is that it is fairly easy to explore these types of questions using simulation.

Here is some R code (other languages would work as well) to simulate and evaluate under your conditions mentioned:

testfun1 <- function(n1=500, n2=4) {
  x1 <- rexp(n1, 1/3)
  x2 <- rexp(n2, 1/3)
  
  # traditional t-test 
  t.p.u <- t.test(x1, x2, var.equal=FALSE)$p.value
  t.p.e <- t.test(x1, x2, var.equal=TRUE)$p.value

  # permutation test
  mu.diff <- mean(x1) - mean(x2)
  x <- c(x1, x2)
  g <- rep(1:2, c(n1, n2))
  out <- replicate(999, {
    tmp.g <- sample(g)
    mean(x[tmp.g==1]) - mean(x[tmp.g==2])
  })
  perm.p <- mean(abs(c(mu.diff, out)) >= abs(mu.diff))
    
  c(t.p.u=t.p.u, t.p.e=t.p.e, perm.p=perm.p)
}

set.seed(1)
out <- replicate(10000, testfun1())
apply(out, 1, function(x) mean(x <= 0.05))

prop.test(sum(out[1,] <= 0.05), 10000)
prop.test(sum(out[2,] <= 0.05), 10000)
prop.test(sum(out[3,] <= 0.05), 10000)

This simulates under the null hypothesis being true and uses an exponential distribution (so CLT very much needed), but you can change the code and run it for yourself to explore other conditions (equal means, but not equal variances; null being false to look at power; etc.).

A properly sized test should give a type I error rate (the result of the apply) of about $0.05$ (since I used that default cut-off). The t-test without assuming equal variances rejects too often (95% C.I. 11.6% - 12.9%), assuming equal variances does not reject often enough (95% C.I. 3.9% - 4.7%), but the permutation test is not statistically distinguishable from 5% (95% C.I. 4.4% - 5.2%).

So for this case the permutation test is "Best", but the assumption of equal variances is not far off (and a little conservative) so may be acceptable (but remember my simulations had equal variances, this could easily change in the equal-mean but unequal variance case). Things could also be different when the true population distribution is less skewed. Try some different simulations to further explore.

The permutation test is somewhat of a "cure all" in that it does not require the assumption of normality. But it is also testing a different null hypothesis than the t-test. It is testing that the 2 distributions are identical (equal variances and shape). When the conditions hold for the t-test to be reasonable, then the t-test will probably also have higher power than the permutation test (you can explore with code similar to the above to see how much different). This is a general phenomenon, tests with more assumptions will tend to have more power when those assumptions are true (the assumptions do give us information), but tests with fewer assumptions can be applied more generally (at the cost of power and precision