2
$\begingroup$

CrossValidated has many discussions on how unequal variances are not a practical issue for two-sample t-tests when Welch correction is used and on how normality assumptions do not play a role (in Type-1-error control) once we have a large enough sample size (some propose a rule of thumb of $n \geq 30$, some $n \geq 50$). The reasoning is that with growing numbers the Central Limit Theorem will ensure normal distribution of the means to be compared.

In those threads I fail to find definitions of $n$. Do these rules of thumb define $n$ as the number of all observations in the test or the number in the smallest group to be tested?

If I had $n_1 = 500$ to compare to $n_2 = 4$ observations. Would the many data on group 1 "heal" the small sample of group 2?

Additional Question: And if not, could I do a permutation test of means on that kind of data? (That seems to be sometimes advised as a cure all but usually, there is not such a thing as a cure all and I may not understand it's limitations).

$\endgroup$
4
  • 2
    $\begingroup$ You shouldn't fuss over the details of any rule of thumb. This rule of thumb is particularly poor, anyway, because most people invoke it only to avoid examining their data. The crucial issue is whether the variation within the second group can be expected to be similar (in distributional shape and size) to the variation within the first group. Often that can be answered through subject-matter knowledge, in which case you can take $n=500+4.$ But when it can't, you are reduced to examining just $n_2=4$ observations, with highly uncertain results, so effectively $n=4$ for this rule of thumb. $\endgroup$
    – whuber
    Commented Jun 29, 2022 at 16:10
  • $\begingroup$ @whuber Thank you for chiming in. I do not fuss over details. My problem is exactly that I do not no the details or whether all the people talking about this assume identical details and if there are rules with different detail, which one to commit to. Albeit sometimes that might be implicit in the explanations which I do not fully understand. Also I do not want to avoid examining my data but sometimes I would like to keep that part small in a publication/discussion short to not distract from the main topic. Isn't that legitimate? (t.b.c.) $\endgroup$
    – Bernhard
    Commented Jun 30, 2022 at 6:13
  • $\begingroup$ @whuber So in short and for dummies, is this what your saying?: If from subject-matter knowledge I assume that distributional shape and size are similar across groups, I can then rely on the overall sample size $N = n_1 + n_2$ being large enough and if not, the smaller sample size $min(n_1, n_2)$ needs to be above some (ill-defined but the same) cut-off? $\endgroup$
    – Bernhard
    Commented Jun 30, 2022 at 6:17
  • $\begingroup$ Not really. I am disputing the general validity of this rule of thumb and suggesting a more informed investigation of the data distribution in its stead. As an example see stats.stackexchange.com/a/69967/919. BTW, I apologize if it sounds like I was suggesting you were among those avoiding examining their data. That was not ever my intention. $\endgroup$
    – whuber
    Commented Jun 30, 2022 at 12:08

1 Answer 1

3
$\begingroup$

One advantage of modern computers (beyond being able to ask questions of people across the world) is that it is fairly easy to explore these types of questions using simulation.

Here is some R code (other languages would work as well) to simulate and evaluate under your conditions mentioned:

testfun1 <- function(n1=500, n2=4) {
  x1 <- rexp(n1, 1/3)
  x2 <- rexp(n2, 1/3)
  
  # traditional t-test 
  t.p.u <- t.test(x1, x2, var.equal=FALSE)$p.value
  t.p.e <- t.test(x1, x2, var.equal=TRUE)$p.value

  # permutation test
  mu.diff <- mean(x1) - mean(x2)
  x <- c(x1, x2)
  g <- rep(1:2, c(n1, n2))
  out <- replicate(999, {
    tmp.g <- sample(g)
    mean(x[tmp.g==1]) - mean(x[tmp.g==2])
  })
  perm.p <- mean(abs(c(mu.diff, out)) >= abs(mu.diff))
    
  c(t.p.u=t.p.u, t.p.e=t.p.e, perm.p=perm.p)
}

set.seed(1)
out <- replicate(10000, testfun1())
apply(out, 1, function(x) mean(x <= 0.05))

prop.test(sum(out[1,] <= 0.05), 10000)
prop.test(sum(out[2,] <= 0.05), 10000)
prop.test(sum(out[3,] <= 0.05), 10000)

This simulates under the null hypothesis being true and uses an exponential distribution (so CLT very much needed), but you can change the code and run it for yourself to explore other conditions (equal means, but not equal variances; null being false to look at power; etc.).

A properly sized test should give a type I error rate (the result of the apply) of about $0.05$ (since I used that default cut-off). The t-test without assuming equal variances rejects too often (95% C.I. 11.6% - 12.9%), assuming equal variances does not reject often enough (95% C.I. 3.9% - 4.7%), but the permutation test is not statistically distinguishable from 5% (95% C.I. 4.4% - 5.2%).

So for this case the permutation test is "Best", but the assumption of equal variances is not far off (and a little conservative) so may be acceptable (but remember my simulations had equal variances, this could easily change in the equal-mean but unequal variance case). Things could also be different when the true population distribution is less skewed. Try some different simulations to further explore.

The permutation test is somewhat of a "cure all" in that it does not require the assumption of normality. But it is also testing a different null hypothesis than the t-test. It is testing that the 2 distributions are identical (equal variances and shape). When the conditions hold for the t-test to be reasonable, then the t-test will probably also have higher power than the permutation test (you can explore with code similar to the above to see how much different). This is a general phenomenon, tests with more assumptions will tend to have more power when those assumptions are true (the assumptions do give us information), but tests with fewer assumptions can be applied more generally (at the cost of power and precision

$\endgroup$
3
  • $\begingroup$ +1 "giva a man a fish-teach a man to fish" in that sense thank you very much for giving me a blueprint R code for doing my own experimentation. Obviously I was hoping that people much smarter then me, and putting in much more time and effort then I ever will, came up with the above rule and could formulate the exact limits within to quickly apply a rule. One additional question: In your permutation code you made shure, that the "original permutation", where nothing is changed, is within the samples is included in the 1000 comparisons abs(c(mu.diff, out)) Any particular purpose in there? $\endgroup$
    – Bernhard
    Commented Jun 30, 2022 at 6:27
  • $\begingroup$ @Bernhard, for a large enough number of permutations it probably will not matter whether we include the original or not. But the definition of p-value usually includes something like "as extreme", and the original data is one possible permutation. So in my opinion, the possibility of computing an exact 0 for a p-value for a permutation test goes against the definition. $\endgroup$
    – Greg Snow
    Commented Jun 30, 2022 at 17:56
  • $\begingroup$ Disallowing $p=0$ is an interesting idea I have not thought of. I am not sure if I have a problem with a Monte Carlo $p$ being exactly zero when I know that Monte Carlo results are not exact, but I will ponder that. $\endgroup$
    – Bernhard
    Commented Jun 30, 2022 at 18:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.