# Behaviour of Welch's t-test with unequal group sizes

It seems that in some cases Welch's t-test can give inflated p-values with imbalanced group sizes. For example, consider the following simulation of unequal group sizes sampled from the same Gaussian distribution.

set.seed(42)
sapply(2:10, function(n) {
out <- replicate(100000, t.test(rnorm(100), rnorm(n))$p.value) mean(out < 0.01) / 0.01 }) # [1] 7.928 3.139 1.913 1.606 1.371 1.285 1.178 1.141 1.192  Is this conclusion correct? Are there general guidelines on group sizes for when a Welch's t-test is appropriate? Are there known ways to correct this apparent bias, or should we in general assume equal variance if groups are imbalanced in terms of size? ### Update: To clarify, I'm interested in establishing minimum sample sizes for a Welch t-test, below which we need to be cautious about the possibility of false positives (restricted to the situation where we are happy to assume equal variances). I think this figure of a simple simulation of samples from the same distribution, with varying sample sizes more cleanly illustrates the issue. In all cases, we expect the mean number of tests with p-value < 0.05 to equal 0.05 if the test performs as expected. I would conclude that we need to be cautious with a Welch t-test if N < ~5 for any group, particularly if N = 2. Note, I've not investigated unequal variance, which as BruceET has pointed out can cause potentially high false positive rates when not accounted for. dosim <- function(var.equal) { sapply(seq_len(nrow(g)), function(i) { pv <- replicate( reps, t.test(rnorm(g[i, "N1"]), rnorm(g[i, "N2"]), var.equal = var.equal)$p.value
)
mean(pv < 0.05)
})
}

reps <- 10000
g <- expand.grid(list(N1 = 2:100, N2 = c(2, 3, 4, 5, 6, 10)))

g$$Welch <- dosim(FALSE) g$$Student <- dosim(TRUE)
g <- tidyr::gather(g, "stat", "value", "Welch", "Student")
g$$N2 <- factor(g$$N2)

ggplot(g, aes(x = N1, y = value, col = N2)) +
geom_point() +
geom_line() +
facet_grid(stat ~ .) +
ylab("Mean(p < 0.05)")


Related:

• Early fussing about with variations leads me to the tentative view that the key sentence in the last link is "The only situation in which the test seems to have undesirable performance is in very small sample sizes." It isn't so much a problem of imbalance of sample sizes as of the very small sizes of the first sample. DF's average about the same as smaller sample size. // Also 1% level is more problematic than 5%.// To use $n_1 = 2$ is to provoke difficulties. Oct 11, 2019 at 19:19
• Hi BruceET. Thanks for your response! I'm particularly concerned about issues with Welch's t-test with small N (< 5). Serious type I error issues occur in some cases (a quick look suggests the issue worsens with imbalanced sizes?). Clearly unequal variances break the assumptions of a pooled t-test (leading the FDR issues). However, serious FDR issues can occur even with equal variances, but when is this the case? Is this worse for imbalanced sizes? Is there a general rule that says we should switch to e.g. Wilcox tests if class sizes are seriously imbalanced (small N < x, large N > y)? Oct 12, 2019 at 1:55
• It would be nice to expand on "To use 𝑛1=2 is to provoke difficulties", particularly given other advice (e.g. stats.stackexchange.com/questions/37993) that "There is no minimum sample size for the t test to be valid". Indeed it is possible to perform 1-sample t-tests. Oct 12, 2019 at 2:01
• In almost any study, using various tests, a samp size of 2 is bound to invite criticism. More particularly, a 2-sample Wilcoxon test needs something like 4 obs. in each sample in order to achieve the 5% level. (Because $2/{8 \choose 4} = 1/35 < 0.05.)$ If you're sure the pop variances are equal, there is no harm, as a technical matter in doing a 2-sample t test with with two very small groups. As a practical matter people will wonder if 2 or 3 observations can actually be representative of anything. A Welch t test almost always has DF $< n_1+n_2 -2;$ w/ DF < 3 the t-dist'n has very fat tails. Oct 12, 2019 at 3:46
• Your point about coherent advice when to use Welch t is well taken. I can't find references right now, but will look. I have always had it in my head from first uses of Welch t that one needs to temper the call for 'universal' use of Welch t with a caveat about very small sample sizes. Your link gives such a caveat, but not all textbook authors do. Oct 12, 2019 at 4:22

This question is sufficiently broad that a comprehensive answer would require a simulation study (some key parts of which have undoubtedly been done), going far beyond our usual style of answers.

A Welch 2-sample t test can't be exactly as good as a pooled t test if we know the two populations have the same variance.

(1) A Welch test at "level 5%" with $$n_1 = 5, n_2 = 100$$ when variances are equal gave Type I error a little above 5% $$(0.054 \pm 0.0014).$$ I didn't explore samples sizes below 5.

set.seed(1234)
out = replicate(10^5, as.numeric(t.test(rnorm(5), rnorm(100))[2:3]))
mean(out[1,])
[1] 4.902672         # avg DF about 5
mean(out[2,] < .05)
[1] 0.05443          # avg P-val about 5%


In the histogram below of 100,000 simulated P-values, the maroon bar at left represents the true significance level (Type I error) of the test. Its width is 0.5 and its height is about 0.108, for a total area of about 0.054,

(2) Pooled test at level 5% with $$n_1 = 5, n_2 = 100$$ when variances are equal. Here, DF = $$n_1 + n_2 - 2 = 103$$ for all tests. The significance level is very near the intended 5%.

set.seed(1234)
out = replicate(10^5, as.numeric(t.test(rnorm(5), rnorm(100), var.eq=T)[2:3]))
mean(out[1,])
[1] 103             # DF exactly n1 + n2 - 2 = 103
mean(out[2,] < .05)
[1] 0.0493          # P-val exactly 5% from transparent theory


(3) However, if the smaller sample has variance 4 and the larger has variance 1, then the pooled tests with a nominal 5% significance level actually has significance level almost 30% (rich in false discoveries).

set.seed(1234)
out = replicate(10^5, as.numeric(t.test(rnorm(5,0,2), rnorm(100), var.eq=T)[2:3]))
mean(out[1,])
[1] 103
mean(out[2,] < .05)
[1] 0.2853


(4) The Welch test may not have exactly 5% Type I error. Even so, with average P-value near 5%, it is clearly preferable to the pooled test when variances are unequal, as in (3).

set.seed(1234)
out = replicate(10^5, as.numeric(t.test(rnorm(5,0,2), rnorm(100))[2:3]))
mean(out[1,])
[1] 4.209818
mean(out[2,] < .05)
[1] 0.05158


(5) First of two simulations focused on power. If variances are unequal and sizes are grossly unbalanced $$(n_1 = 5, n_2 =100$$), the power to detect that $$\mu_1 = 4$$ differs from $$\mu_2 = 0$$ is about 90%.

set.seed(1234)
out = replicate(10^5, as.numeric(t.test(rnorm(5,4,2), rnorm(100))[2:3]))
mean(out[1,])
[1] 4.209818
mean(out[2,] < .05)
[1] 0.90977


In the figure below, the green bar at left represents the power of the test. So now, a tall first bar is a good thing.

(6) If we balance the data (both samples of size 5), the power of the Welch test is reduced very little if any, compared to the previous simulation. Changing the means to be equal, leaving sample sizes at 5, and leaving variances unequal, a similar simulation (not shown) gave significance level 4.9%

set.seed(1234)
out = replicate(10^5, as.numeric(t.test(rnorm(5,4,2), rnorm(5))[2:3]))
mean(out[1,])
[1] 6.000759
mean(out[2,] < .05)
[1] 0.89743


Summary comments. None of these simulations changes the advice to prefer the Welch test to the pooled test, except for tiny sample sizes, In that case prior information whether variances are equal might help to decide between Welch test and a pooled test.

I have tried to probe in relevant directions with the few simulations above, but I certainly don't claim that they settle anything. If anyone has information from related simulations that would refine or extend my 'no new news' conclusions here, I'd be happy to see them.

The issue here does not appear to be due to imbalanced sample sizes per se, but rather, non-uniformity of the p-value seems to be occurring due to using very small sample sizes for one of the samples. Below I show high-resolution histograms showing simulated p-values for a comparison of one sample with $$n_X = 100$$ standard normal values and another sample with $$n_Y = 2,...,10$$ standard normal values. (Each comparison uses $$K = 10^6$$ simulations, and the histograms use bin widths $$w = 0.01$$ so you get high-resolution for the true distributions.)

As you can see, for very small samples there is a non-uniform distribution with a higher than expected probability of a small p-value. I am not entirely certain where this phenomena originates, but I have some strong suspicions. It is well-known that the standard sample variance estimator yields an estimated standard deviation that is biased for small samples (see e.g., here). For normal data the sample standard deviation is biased downward by a known "correction factor" that has quite a heavy effect for very small samples. If one were to underestimate the true standard deviation of the smaller sample in Welch's T-test, then the natural expectation would be that this would underestimate the likely different between sample means under the null hypothesis of no difference, which would bias the p-value downward. Since this is exactly what we see in the histograms, my suspicion is that this phenomena occurs due to the downward bias of sample standard deviations as an estimate of true standard deviations. I suspect that if you were to apply the standard small sample "corrections" to the sample standard deviations, in the test statistic for Welch's test, then this phenomena would be substantially ameliorated.

#Generate p-value simulations from Welch's test
#First sample is set to sample size of 100
#Second sample has sample sizes from 2, ..., 30
set.seed(97142903)
SIMS  <- 10^6;
PVALS <- matrix(NA, nrow = SIMS, ncol = 9);
for (n in 2:10)   {
for (i in 1:SIMS) {
PVALS[i, n-1] <- t.test(rnorm(100), rnorm(n))$p.value; } } #Generate data frame of p-value simulations DATA <- data.frame(n = rep(2:10, each = SIMS), p = as.vector(PVALS)); #Plot the KDEs of the p-value simulations library(ggplot2); THEME <- theme(plot.title = element_text(hjust = 0.5, size = 14, face = 'bold'), plot.subtitle = element_text(hjust = 0.5, face = 'bold')); FIGURE <- ggplot(aes(x = p), data = DATA) + geom_histogram(binwidth = 0.01, boundary = 0, fill = 'blue') + facet_wrap(~ n, ncol = 1) + THEME + ggtitle('Histograms of p-value simulations') + labs(subtitle = '(Sample of 100 against sample of stated size)') + xlab('p value') + ylab('Count'); FIGURE;  • Thanks Ben. I come back to this issue that the standard recommendation is that there is no minimum sample size for a t test. Indeed I think the t-test is originally designed to handle small n? It seems that this is showing (in some cases perhaps) we should be very cautious about using Welch’s t-test with small sample sizes? Consistent with my own quick simulations, it seems when small n = 2 there is strong potential for false positives. Oct 12, 2019 at 3:23 • If you want to deal with a very small sample size, you might try correcting the sample standard deviation using the standard correction terms. This might be sufficient to ensure that Welch's T-test runs well on small samples. – Ben Oct 12, 2019 at 3:42 • I don't think bias of$S$for small$n$has anything to do with behavior of pooled t. The t dist'n involves$(n-1)S^2/\sigma^2 \sim Chisq(n-1).$However, I'm not sure if that bias creeps into the formula for Welch df. // Nice simulation (+1):$10^6$trumps$10^5,\$ but I was running out of time and got impatient with longer run times on my geriatric machine. Oct 12, 2019 at 4:11
• I thought you may like to know that your hypothesis turns out not to be correct. The reason is nothing to do with the underestimation of the standard deviation for small samples, as you can find by simulation after applying the correction (it inappropriately biases the result). The actual reason is that the Welch t-test is just an approximation of the Welch distribution using the Student's t-distribution, which assumes the central limit theorem and does not capture the tails well in all cases. The Welch distribution is hard to work with directly (e.g., no closed form expression). Dec 29, 2020 at 21:20
• @OleksandrR.: That is interesting. In that case, may I recommend running a new simulation in which you fix a low sample size but vary the underlying distribution over a range of distributions that vary from leptokurtic to mesokurtic (with the normal distribution in the middle). That way you can examine the effect of the underlying kurtosis on the test over small samples.
– Ben
Dec 29, 2020 at 21:25