I am comparing the anatomy of two Neanderthal populations for which sample sizes, the number of available bones, is low (usually between 3 and 8 for a given trait). I am using Welch's unequal variance t-test because I have no reason to believe that the populations' variances should be equal.

Considering traits such as brain size, stature, the length of the femur and of the humerus, I am using a practice dataset (for males, n usually ~500) collected from 20th century Americans to examine the assumption of normality. Given sample sizes of, say, 3 and 3 for the two Neanderthal populations, as usefully suggested by Glen_b, I sample a very large number of times 3 and 3 data points from the practice dataset, run a t test on these necessarily identically distributed groups of data points, and evaluate the proportion of such tests that spuriously fall under the alpha threshold of significance. I do not remove outliers from the practice dataset, even if it sometimes looks like the team that collected the data accidentally recorded a child's bones in the adult data set:

enter image description here

If about 10% of p-values fall under 0.10 and 1% under 0.01, I will feel reasonably entitled to assume the data are sufficiently close to being normally distributed to run a t-test on the Neanderthal data points.

It turns out that with the unequal variance t-test almost all traits I have tested with the practice data are slightly to significantly more conservative than expected under normality. This is also the case with equal variance but the effect is much less pronounced. For the humerus (the long bone between the shoulder and elbow), after resampling and t-testing two groups, both of size 3, with unequal variance 100,000 times, 0.48% of p-values were under 0.01 and 7.52% were under 0.10. When I tested with equal variance, 0.95% of p-values were under 0.01 and 9.70% under 0.10. Histograms of p-values look like this, unequal variance on the left, equal variance on the right:

enter image description here

One paragraph of Wikipedia's article on Welch's t-test reads:

Welch's t-test is more robust than Student's t-test and maintains type I error rates close to nominal for unequal variances and for unequal sample sizes under normality.

Why might it not be the case here? Is Welch more sensible to outliers or deviations from normality? Is Welch's Type I error rate not as close to nominal as the equal variance's error rate when sample sizes are very small?

EDIT: This seems to have something to do with the sample size. If instead of resampling 3 and 3 data points 100,000 times, I artificially do so for 12 and 12 data points, then for unequal variance 0.86% of p-values are under 0.01 and 9.78% are under 0.10, whereas for equal variance 0.92% of p-values are under 0.01 and 10.1% are under 0.10, with histograms that now look like this:

enter image description here

  • 1
    $\begingroup$ In the end this conservatism might not be a bad thing if it punishes the use of very small samples that make the test low-powered. Still, I don't understand where it's coming from. $\endgroup$
    – Pertinax
    Commented Sep 3, 2017 at 14:41

1 Answer 1


The question is a bit of a moot point, because the frequentist analysis assumes one of two things: the variances are exactly equal and the distributions are normal, or the variances may be arbitrarily different and the distributions are normal. Neither of these is a reasonable assumption. That's why a Bayesian t-test is preferred. It allows for non-normality while favoring normality, and allows for unequal variances while favoring a variance ratio that is not too far from 1.0. As N gets larger these favoritisms vanish. All uncertainties are taken into account and exact inference is obtained for any sample size.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.