As part of my homework, I've got this question. Can someone confirm whether this statement is true or false? To help me understand I would prefer if an explanation is also given.
"The two-sample t-test with unequal variances has, as its null hypothesis, that the variances of the two populations involved are the same."
You're talking about a pooled 2-sample t test, of $H_0: \mu_1 = \mu_2$ vs $H_a: \mu_1 \ne \mu_2.$ This test assumes that $\sigma_1 = \sigma_2.$
Let's consider a sample of size $n_1 = 10$ from $\mathsf{Norm}(\mu = 50, \sigma_1 = 1)$ and a sample of size $n_2 = 40$ from $\mathsf{Norm}(\mu = 50, \sigma_1 = 1).$ That is, the two sample means are equal. We reject $H_0$ at the 5% level, if the P-value $< 0.05.$
Comparing two specific such samples, what output do we get from the pooled 2-sample t test?
set.seed(1234)
x1 = rnorm(10, 50, 1); x2 = rnorm(40, 50, 1)
t.test(x1, x2, var.eq=T)
Two Sample t-test
data: x1 and x2
t = 0.27657, df = 48, p-value = 0.7833
alternative hypothesis:
true difference in means is not equal to 0
...
sample estimates:
mean of x mean of y
49.61684 49.52947
All is well. From the simulation, we know that $\mu_1 - \mu_2 = 50.$ (Also that $\sigma_1^2 = \sigma_2^2 = 1.)$ And the test has (correctly) failed to reject $H_0.$
However, 5% of the time, a pooled test at the 5% level will make a mistake, rejecting $H_0$ with a P-value $ < 0.05.$ We could discuss the theory to show that this rejection rate is correct. Instead, let's look at actual results of a million such pooled 2-sample t tests.
set.seed(817)
pv = replicate(10^6,
t.test(rnorm(10,50,1), rnorm(40,50,1), var.eq = T)$p.val)
mean(pv <= 0.05)
[1] 0.049801
Just 'as advertised': The pooled 2-sample t test has incorrectly rejected $H_0$ in almost exactly 5% of the tests on one million sets of two samples from the designated distributions.
Now let's see what happens if we keep everything exactly the same--except that we change the population variances to be unequal, with $\sigma_1^2 = 16$ and $\sigma_2^2 = 1.$
set.seed(818)
pv = replicate(10^6,
t.test(rnorm(10,50,4), rnorm(40,50,1), var.eq = T)$p.val)
mean(pv <= 0.05)
[1] 0.293618
Now the test is falsely rejecting about 30% of the time---much more than 5% of the time. The 'null distribution' (distribution when $H_0$ is true) has changed substantially. Obviously, the change from equal variances to unequal variances has made a difference in how the pooled t test works. The t test cannot have "detected" that means are unequal, because they aren't. Maybe it is unfair to say that the test has "detected" unequal variances, but it is clear that unequal variances do change how the test performs.
One can quibble whether equal variances are part of the null hypothesis. But, using the pooled t test, equal variances are essential to a fair test of the null hypothesis.
Notes about R code: (a) The default 2-sample t test in R is the Welch test, which does not assume equal variances. The parameter var.eq=T leads to use of the pooled test. If one uses the Welch test for samples from populations with
unequal variances, the significance level is very nearly 5%.
set.seed(819)
pv = replicate(10^6,
t.test(rnorm(10,50,4), rnorm(40,50,1))$p.val)
mean(pv <= 0.05)
[1] 0.050252
(b) The vector pv contains P-values of a million pooled tests. The logical vector pv <= 0.05 contains a million TRUEs and FALSEs.
The mean of a logical vector is the proportion of its TRUEs.
(c) The comprehensive text An intro. to statistical methods and data analysis, 7e, by Ott and Longnecker (2016), Cengage, has a useful table of the critical values of the pooled t test for various sample sizes and ratios of $\sigma_1/\sigma_2,$ Table 6.4, p311. Tabled values are based on fewer iterations than used in this Answer, so they do not agree exactly with answers here. (In particular, all tabled values in the column for $\sigma_1/\sigma_2 = 1$ should be exactly 0.050.)
No it’s false: you are still testing the hypothesis that the mean is different across the two groups. That’s it. You are not testing the homogeneity of variance. The difference between the case where you assume equal variance or different variance in the two groups is not the null hypothesis of the test: what actually change between the two cases is only your a-propri assumption about the variance in the population of the two groups, that determines the way you compute the standard error to be used at the denominator in the t statistic. But this is NOT what you are testing: this is what you are assuming to define the distribution of the t statistic. See this link for further details.
