One advantage of modern computers (beyond being able to ask questions of people across the world) is that it is fairly easy to explore these types of questions using simulation.
Here is some R code (other languages would work as well) to simulate and evaluate under your conditions mentioned:
testfun1 <- function(n1=500, n2=4) {
x1 <- rexp(n1, 1/3)
x2 <- rexp(n2, 1/3)
# traditional t-test
t.p.u <- t.test(x1, x2, var.equal=FALSE)$p.value
t.p.e <- t.test(x1, x2, var.equal=TRUE)$p.value
# permutation test
mu.diff <- mean(x1) - mean(x2)
x <- c(x1, x2)
g <- rep(1:2, c(n1, n2))
out <- replicate(999, {
tmp.g <- sample(g)
mean(x[tmp.g==1]) - mean(x[tmp.g==2])
})
perm.p <- mean(abs(c(mu.diff, out)) >= abs(mu.diff))
c(t.p.u=t.p.u, t.p.e=t.p.e, perm.p=perm.p)
}
set.seed(1)
out <- replicate(10000, testfun1())
apply(out, 1, function(x) mean(x <= 0.05))
prop.test(sum(out[1,] <= 0.05), 10000)
prop.test(sum(out[2,] <= 0.05), 10000)
prop.test(sum(out[3,] <= 0.05), 10000)
This simulates under the null hypothesis being true and uses an exponential distribution (so CLT very much needed), but you can change the code and run it for yourself to explore other conditions (equal means, but not equal variances; null being false to look at power; etc.).
A properly sized test should give a type I error rate (the result of the apply
) of about $0.05$ (since I used that default cut-off). The t-test without assuming equal variances rejects too often (95% C.I. 11.6% - 12.9%), assuming equal variances does not reject often enough (95% C.I. 3.9% - 4.7%), but the permutation test is not statistically distinguishable from 5% (95% C.I. 4.4% - 5.2%).
So for this case the permutation test is "Best", but the assumption of equal variances is not far off (and a little conservative) so may be acceptable (but remember my simulations had equal variances, this could easily change in the equal-mean but unequal variance case). Things could also be different when the true population distribution is less skewed. Try some different simulations to further explore.
The permutation test is somewhat of a "cure all" in that it does not require the assumption of normality. But it is also testing a different null hypothesis than the t-test. It is testing that the 2 distributions are identical (equal variances and shape). When the conditions hold for the t-test to be reasonable, then the t-test will probably also have higher power than the permutation test (you can explore with code similar to the above to see how much different). This is a general phenomenon, tests with more assumptions will tend to have more power when those assumptions are true (the assumptions do give us information), but tests with fewer assumptions can be applied more generally (at the cost of power and precision