5
$\begingroup$

Following my previous (still unanswered) question, I'm trying to find what happens when we construct confidence interval for means difference using Student's t-test (for one or two samples), assuming that variances $\sigma_x^2$ and $\sigma_y^2$ for two populations $X \sim \mathcal{N}(\mu_x, \sigma_x^2)$ and $Y\sim \mathcal{N}(\mu_y, \sigma_y^2)$ are equal, when they really are not.

I know that in this case we simply use Welch t-test and it works, but I want to understand why.

Say we know our differences are equal and want to find confidence interval for means difference. Then assuming null hypothesis $H_0: \mu_x = \mu_y$ we can just merge two samples into one and find $[\bar{X}_n - t_{\alpha, n-1}\frac{S_n}{\sqrt{n}}, \bar{X}_n + t_{\alpha, n-1}\frac{S_n}{\sqrt{n}}]$, where $S_n$ is the estimated sample variance. However, remember the fact that variances are not equal, so it means we can not perform such a test and need to perform Welch t-test with this statistic:

$$t = \frac{\bar{X}_n - \bar{Y}_m}{\sqrt{\frac{\sigma_x^2}{n} + \frac{\sigma_y^2}{m} }}.$$

Also I believe we could use Student's t-test for two samples instead of one.

Question is: what problems occur using Student's test instead of Welch's with $t$-statistic described above?

$\endgroup$
3
  • 3
    $\begingroup$ I think asking a good question requires you go through a bit more thinking and synthesizing before you type it up. In both this question and the one you link to, which, as you correctly note, no one has answered yet, people have stated that they were not sure what you were asking. I tend to learn a lot in the process of posting my questions and sharpening them; sometime towards the end of typing up the question, having thought about it some more I realize that I've answered my own question in the process of organizing my thoughts and pinpointing the source of my confusion. I hope this helps. $\endgroup$ Apr 15, 2022 at 19:49
  • 1
    $\begingroup$ Can your question be formulated as: Why do we construct null hypotheses where of no difference in means and no difference in variance instead of always just assuming different variances? I'm not sure exactly what you are driving at with the question. $\endgroup$ Apr 16, 2022 at 0:43
  • 1
    $\begingroup$ The Welch test does follow the t distribution under $H_0,$ using a number of degrees of freedom that (very nearly) corrects for unequal variances. $\endgroup$
    – BruceET
    Apr 16, 2022 at 9:44

2 Answers 2

9
$\begingroup$

What problems causes using Student's tests instead of Welch's with t-statistic described above?

Let's back up a bit and rephrase. By "using Student's tests" what you are really saying is that in the case in which the variances $\sigma_x^2$ and $\sigma_y^2$ are unknown but can be assumed to be equal, a certain statistic $\frac{\bar{X}-\bar{Y}-(\mu_X-\mu_Y)}{S_p \sqrt{\frac{1}{n}+\frac{1}{m}}}$ follows the t-distribution with n+m-2 degrees of freedom ($S_p^2$=$\frac{(n-1)S_X^2+(m-1)S^2_Y}{n+m-2}$ is the pooled variance). That means we know the sampling distribution of this statistic and that is why we can contextualize probabilistically the single realized value of this statistic in our sample ($\bar{X}-\bar{Y}$) and draw an inference from it to the population parameter of interest ($\mu_x-\mu_Y$). Why does $\frac{\bar{X}-\bar{Y}-(\mu_X-\mu_Y)}{S_p \sqrt{\frac{1}{n}+\frac{1}{m}}}$ follow a t-distribution?

$\frac{\bar{X}-\bar{Y}-(\mu_X-\mu_Y)}{S_p \sqrt{\frac{1}{n}+\frac{1}{m}}}$=$\frac{\frac{\bar{X}-\bar{Y}-(\mu_X-\mu_Y)}{\sigma\sqrt{\frac{1}{n}+\frac{1}{m}}}}{{\sqrt{\frac{S_p^2}{\sigma^2}}}}$

The numerator has a standard normal distribution, while the denominator can be shown to have a $\chi^2$ distribution with $n+m-2$ degrees of freedom. Because the numerator and denominator are independent, we conclude (from the definition of the t-distribution) that their ratio is t distributed with $n+m-2$ degrees of freedom. Why does the numerator have the standard normal distribution? Because $\bar{X}-\bar{Y}$ is Normally distributed with mean $\mu_X-\mu_Y$ and variance $\frac{\sigma^2}{n}+\frac{\sigma^2}{m}=\sigma^2(\frac{1}{n}+\frac{1}{m})$. This last equality would not hold if $\sigma_x^2$ and $\sigma_y^2$ are not equal, which means that the $\sigma's$ in the above rightmost fraction do not cancel out; we cannot get from the right expression to the left one (which does not have any $\sigma's$).


So what does that mean?

If the variances $\sigma_x^2$ and $\sigma_y^2$ are unknown and cannot be assumed to be equal, the statistic $\frac{\bar{X}-\bar{Y}-(\mu_X-\mu_Y)}{S_p \sqrt{\frac{1}{n}+\frac{1}{m}}}$ does not follows the t-distribution. Can we find another statistic that has a well-behaved sampling distribution, maybe even one we've already catalogued? We've been unsuccessful at it thus far and not for the lack of trying. It is known as the Behrens-Fisher problem. B.L. Welch came up with an approximate solution: the Welch statistic. The Welch statistic is approximately distributed as a $t$ random variable. Why approximately distributed and not exactly distributed? Because the Welch statistic is the ratio of a standard normal random variable and a variable that is not exactly but only approximately $\chi^2$ distributed.

$\endgroup$
1
  • 1
    $\begingroup$ That obvious that this statistic does not follows t-distribution in the case of unequal variances, isn't it? Question was about problems, i.e. what happens in practice. But thanks for mentioning B-F problem! $\endgroup$
    – taciturno
    Apr 15, 2022 at 19:27
4
$\begingroup$

At the very end of your long question, you get around to the point, asking why not go ahead and use the two-sample pooled test, even when variances are not known to be equal, instead of using the Welch test. Here's why not:

Suppose you have a sample of size 10 from a normal distribution with SD $\sigma_1 = 4$ and a sample of size 40 from a normal distribution with $\sigma_2 = 1.$ Also suppose the two sample means are equal: $\mu_1=\mu_2 = 50$ (so, $H_0$ is true) and that we want a test at the 5% level.

The following simulation in R shows the very bad behavior of the pooled 2-sample t test. (Note parameter var.eq=T to get the pooled test.)

set.seed(2022)
pv = replicate(10^5, t.test(rnorm(10,50,4),
             rnorm(40,50,1), var.eq=T)$p.val)
mean(pv <= 0.05)
[1] 0.2948

Instead of the expected 5% significance level (Type I Error), we have significance level about 30%. That is a massive false discovery rate.

By contrast, the significance level using the default Welch test in the t.test procedure, we get very nearly the nominal 5% significance level. Although it uses an approximation, the Welch test takes the unequal variances into account to give very nearly the correct results.

set.seed(2022)
pv = replicate(10^5, t.test(rnorm(10,50,4),
             rnorm(40,50,1))$p.val)
mean(pv <= 0.05)
[1] 0.05084

It is as simple as that. The pooled test gives incorrect and misleading answers and the Welch test gives correct answers.

$\endgroup$
1
  • 2
    $\begingroup$ +1: Nice answer, Bruce. The simulation lets us see what goes wrong here, maybe more so than understand what is going wrong, but it is certainly eye-opening to see what the problem is. $\endgroup$ Apr 16, 2022 at 20:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.