Welch test seems to perform much worse than equal variance t-test

Question

The SciPy function in Python, ttest_ind() by default works with the $t$-test that assumes equal variances. There is a parameter, equal_var = False that switches it to the Welch test where equal variances in the two samples are not assumed.

This seems to suggest that the Welch test should perform better when the two samples have different variances by design. So, I set out to put this to the test. Surprisingly, the conclusion I got was the exact opposite.

from scipy.stats import norm, ttest_ind 
a1 = norm.rvs(10, 14, size = 6)
a2 = norm.rvs(13, 3, 100)
p_val1 = ttest_ind(a1, a2)[1]
p_val2 = ttest_ind(a1, a2, equal_var = False)[1]

Here, we generate 6 samples from a normal distribution with mean 10 and standard deviation 14. Then 100 samples from another normal distribution with mean 13 and standard deviation 3. It's pretty clear that the two samples have unequal variance. The first $p$-value is from the simple $t$-test that assumed equal variance while the second one is from the Welch test. The first $p$-value is consistently less than 1% while the second one is generally about 30-40%. And since the means are in-fact different, the Welch test is underperforming. One critique here is that I didn't consider the false positive rate, only power. This is rectified in the plot below, which plots the false positive to false negative rates of the two tests, hence considering both.

This can be visualized in the alpha-beta profile of the two tests (false positive rate plotted with false negative rate). The two-sample $t$-test has much lower false negative rates than the Welch test (plotted in blue).

Why is the Welch test beaten so handily? Are there other conditions where it might outperform the two sample $t$-test? If not, why ever use it?

---

EDIT: The plot below had a bug. The actual performance of the two tests is the same in terms of statistical power. See my answer for the corrected plot.

---

Answer 1

You are using a flawed metric for the performance of the test by focusing only on the power to detect a difference, but ignoring the possibility of falsely rejecting the null hypothesis. If you just want to optimize this metric the "best" statistical test for you to use would be one that just returns p = 0 for all input data.

You should replicate your example with the means in both samples set to 0.

As you can see from the below example case if there is no difference the normal t-test rejects the null hypothesis wrongly about 50% of the time.

n.sim <- 1e4
p1.w <- p2.t <- rep(0, n.sim)
for (i in 1:n.sim)
{
  x <- rnorm(6, 0, 13)
  y <- rnorm(100, 0, 3)
  p1.w[i] <- t.test(x,y, var.equal = FALSE)$p.value
  p2.t[i] <- t.test(x,y, var.equal = TRUE)$p.value
} 
sum(p1.w < 0.05)/n.sim
sum(p2.t < 0.05)/n.sim

The normal t.test suffers the most from violation of the assumption of equal variance if the sample sizes are unequal. If the smaller sample has higher variance it will falsely reject the null hypothesis too often & if the smaller sample size has lower variance it is too conservative. Your example is of the first case.

Answer 2

This is a concern about the power of the Welch Test as compared to the textbook t test. Let's set up little simulation experiment and see if there are any discrepancies.

The things that we can modulate are:

• The difference in means
• The sample size
• The difference in variance between groups, and
• If we use welch's or the regular t test.

I'm going to modulate the sample size by examining $n_1/n_2$. Same with the variance $\sigma_1/\sigma_2$. I'm going to simulate results over a grid and run 1000 tests for each grid point. I'm doing this in R, so I'm assuming the scipy devs did not implement the test incorrectly.

Here is the R code for the experiment

library(tidyverse)


pwr = function(n, effect_size, ratio, equal_var = F){
  reps = replicate(1000,{
    x = rnorm(100, 0, 1)
    y = rnorm(ceiling(n*100), effect_size, ratio)
    test = t.test(x,y,var.equal = equal_var)$p.value
    test<0.05
  })
  
  mean(reps)
}

results = crossing(
  n = seq(0.1, 1, 0.1),
  effect_size = c(0.2, 0.5, 0.8, 2.0),
  ratio = c(1, 2, 4, 8, 16),
  equal_var = c(T,F)
) %>% 
  mutate(power = pmap_dbl(list(n=n, effect_size = effect_size, ratio = ratio, equal_var = equal_var), pwr))


results %>% 
  ggplot(aes(x=n, y=power, color = factor(equal_var)))+
  geom_line()+
  facet_grid(ratio~effect_size, labeller = label_both)+
  scale_y_continuous(labels = scales::percent)

Results

I can replicate some results in so far as the Welch test seems to have smaller power in some circumstances, even when the assumptions are met. You can see that as the ratio of $n_1/n_2$ approaches 1 (i.e. the groups are the same size) the two tests are more or less equivalent. Note that the curves should be increasing to the right in all cases (except the effect size = 0 case). However, when there is a drastic difference in sample size (as in your example), the t test has superior power. When the variance are in fact equal, the power is comparable.

There is something else interesting here. Take a look a the effect_size:0 column. In this column, there is no difference in groups truly (i.e. they have the same population mean). Any rejection under this column is a false positive. Ideally, both the t test and the welch's t test should have a statistical power of 0, and a false positive rate of 0.05. You can see that the t test is rejecting the null when it should not be. I think that is because the sample sizes are small, hence the means have a large sampling variance leading to a rejection of the null. I think this is further supported because we see the "power" in these cases decrease as the sample size increases. More samples mean more precise estimates of the mean, which should result in fewer false positives.

This is interesting, and I'm going to have to come back to this tomorrow because it is Valentines day and my girlfriend doesn't like me doing stats too late.

Answer 3

Inspired by the answer from @DemetriPananos and created a similar plot for various ratios of $\frac{n_1}{n_2}$ and $\frac{\sigma_1}{\sigma_2}$. These plots have the actual false positive rate on the x-axis and false negative rate on the y-axis for the two tests. In terms of statistical power, the two tests are near identical.

The code for the plot below is at this gist: https://gist.github.com/ryu577/ed535a6457dc98672c39cfe47f1894b6

The effect size between the null and alternate is always the same (3).

In addition, let's look at the type-1 error rate (or the significance threshold) to false positive rate profile. The plots below show as the other answers indicate that the Welch test is able to control the false positive rate while for the equal variance t-test, the false positive rate diverges from a straight line (meaning the p-values are not uniform).