# Under what conditions is Welch's unequal variance t-test more conservative than Student's equal variance t-test?

I am comparing the anatomy of two Neanderthal populations for which sample sizes, the number of available bones, is low (usually between 3 and 8 for a given trait). I am using Welch's unequal variance t-test because I have no reason to believe that the populations' variances should be equal.

Considering traits such as brain size, stature, the length of the femur and of the humerus, I am using a practice dataset (for males, n usually ~500) collected from 20th century Americans to examine the assumption of normality. Given sample sizes of, say, 3 and 3 for the two Neanderthal populations, as usefully suggested by Glen_b, I sample a very large number of times 3 and 3 data points from the practice dataset, run a t test on these necessarily identically distributed groups of data points, and evaluate the proportion of such tests that spuriously fall under the alpha threshold of significance. I do not remove outliers from the practice dataset, even if it sometimes looks like the team that collected the data accidentally recorded a child's bones in the adult data set:

If about 10% of p-values fall under 0.10 and 1% under 0.01, I will feel reasonably entitled to assume the data are sufficiently close to being normally distributed to run a t-test on the Neanderthal data points.

It turns out that with the unequal variance t-test almost all traits I have tested with the practice data are slightly to significantly more conservative than expected under normality. This is also the case with equal variance but the effect is much less pronounced. For the humerus (the long bone between the shoulder and elbow), after resampling and t-testing two groups, both of size 3, with unequal variance 100,000 times, 0.48% of p-values were under 0.01 and 7.52% were under 0.10. When I tested with equal variance, 0.95% of p-values were under 0.01 and 9.70% under 0.10. Histograms of p-values look like this, unequal variance on the left, equal variance on the right:

One paragraph of Wikipedia's article on Welch's t-test reads:

Welch's t-test is more robust than Student's t-test and maintains type I error rates close to nominal for unequal variances and for unequal sample sizes under normality.

Why might it not be the case here? Is Welch more sensible to outliers or deviations from normality? Is Welch's Type I error rate not as close to nominal as the equal variance's error rate when sample sizes are very small?

EDIT: This seems to have something to do with the sample size. If instead of resampling 3 and 3 data points 100,000 times, I artificially do so for 12 and 12 data points, then for unequal variance 0.86% of p-values are under 0.01 and 9.78% are under 0.10, whereas for equal variance 0.92% of p-values are under 0.01 and 10.1% are under 0.10, with histograms that now look like this:

• In the end this conservatism might not be a bad thing if it punishes the use of very small samples that make the test low-powered. Still, I don't understand where it's coming from. Commented Sep 3, 2017 at 14:41