0
$\begingroup$

In unequal variance t-test (Welch t-test):

$$H_0 = \text{No difference in means, but variance can differ}$$ $$H_1 = \text{Two sample means are significantly different}$$

I don't see the point of unequal variance test. Even though sample means are the same, but if the variance is different, what does it tell us?

Please address this question with the following case studies.

Case 1: two different medical procedure was applied on the same group of patients. How to test if two procedures are significantly different from each other?

Case 2: one class taught by the same teacher is split into two groups and take exams. But the supervisor who has the exam result doesn't know about this. He wants to know if the two groups (samples) came from the same classroom (population). What does unequal variance test do here?

I also read that F-test is used to test difference in variance. How does F-test relate to unequal, or equal variance test?

$\endgroup$
  • 1
    $\begingroup$ Your statement of the hypotheses for the Welch test is not correct --- alternative hypothesis should not refer to a "significant difference". $\endgroup$ – Reinstate Monica Jun 17 at 1:36
  • 1
    $\begingroup$ In practice, for both Case 1 and Case 2, the main issue is almost always whether the normal population means differ. (If you really want to know whether the two populations are different in any way, there tests for that, but not pooled t, Welch t, nor F-test.) $\endgroup$ – BruceET Jun 17 at 5:02
  • $\begingroup$ @BruceET could you throw me with some names or good stackexchange link for techniques used to test if two populations are different, and quantify the extent of difference? $\endgroup$ – Eric Kim Jun 17 at 5:25
  • $\begingroup$ Will look around tomorrow. Past my bedtime now. Kolmogorov-Smirnov test is one; ks.test in R. Also, Wilcoxon rank sum (2-sample) test; wilcox.test in R. You might look at Wikipedia and search on those names as a start. Recent issue (within past yr) of The American Statistician had a short article on a test of whether normal means and/or variances differ. (Kind of messy and real-life examples seemed weakly motivated to me at first reading.) $\endgroup$ – BruceET Jun 17 at 5:38
2
$\begingroup$

The alternative to the Welch 2-sample t test is the pooled 2-sample t test. In order for the pooled test to give reliable results, it is necessary for population variances to be equal. But the Welch test works well--whether or not the variances are equal.

Pooled t test. If I have a sample of size 10 from $\mathsf{Norm}(\mu = 50, \sigma=8)$ and a sample of size 30 from $\mathsf{Norm}(\mu = 50, \sigma=8),$ then the pooled 2-sample t test (with a critical value chosen for level $\alpha = 0.05)$ has probability 5% of rejecting $H_0: \mu_1 = \mu_2$ vs $H_a: \mu_1 \ne \mu_2.$ This is as it should be for a test at the 5% level of significance.

set.seed(615)  # means equal, variances equal
pv = replicate(10^5, t.test(rnorm(10,50,8), rnorm(30,50,8), var.eq=T)$p.val )
mean(pv < .05)
[1] 0.0501     # as should be

However, if I have a sample of size 10 from $\mathsf{Norm}(\mu = 50, \sigma=8)$ and a sample of size 30 from $\mathsf{Norm}(\mu = 60, \sigma=8),$ then the pooled 2-sample t test has a high probability of rejecting $H_0: \mu_1 = \mu_2$ vs $H_a: \mu_1 \ne \mu_2.$ In the simulation below we see that this probability, called the 'power', is about 92%.

set.seed(616)  # mean unequal, variances equal
pv = replicate(10^5, t.test(rnorm(10,50,8), rnorm(30,60,8), var.eq=T)$p.val )
mean(pv < .05)
[1] 0.91576    # very good power

So the pooled t test works well when variances are known to be equal.

But what happens if the means are equal and the variances are unequal with $\sigma_1 = 10$ in the first population and with $\sigma_2 = 5$ in the second population?

Then what ought to be a test at the 5% level has become a test at about the 15% level. So I'll falsely believe means are unequal when they really are equal. As a result, I might publish some false "discoveries."

set.seed(617)  # mean equal, variances unequal
pv = replicate(10^5, t.test(rnorm(10,50,10), rnorm(30,50,5), var.eq=T)$p.val )
mean(pv < .05)
[1] 0.15408    # excessively high probability of Type I error

Welch t test. By contrast, the Welch test uses a modified t statistic, (usually) with a smaller number of degrees of freedom, in order to get a test close to the 5% level. [Note that in the R procedure t.test, removing the argument var.eq=T changes the procedure from a pooled to a Welch test.]

set.seed(618)  # Welch with mean equal, variances unequal
pv = replicate(10^5, t.test(rnorm(10,50,10), rnorm(30,50,5))$p.val )
mean(pv < .05)
[1] 0.05169    # as it should be

Moreover, the Welch test still does a pretty good job of detecting when means are unequal: it has power about 79%.

set.seed(619)  # Welch with mean unequal, variances unequal
pv = replicate(10^5, t.test(rnorm(10,50,10), rnorm(30,60,5))$p.val )
mean(pv < .05)
[1] 0.78657    # reasonably good power

What's the point? In conclusion, the point of using the Welch test is that performs well even if population variances are not equal. In practice, one usually doesn't know whether or not population variances are equal. So good statistical practice is to use the Welch version of the two-sample t test, unless one has reliable prior evidence that population variances are equal.

Note: The F-test for unequal variances has poor power. It should not be used to 'screen' whether to use the pooled or the Welch test. If there is any uncertainty about unequal variances, automatically use the Welch test.

$\endgroup$
1
$\begingroup$

Your question arises from a confusion between the null hypothesis of a test ($H_0$) and the assumptions of a test:

  • $H_0$ is a default statement that there is "nothing special" (no correlation, no difference, etc.). Then the p-value is the probability to observe the data, or something more extreme, assuming $H_0$ is true. Both the Student's $t$-test and the Welch $t$-test share the same $H_0$: that two populations have equal means
  • the assumptions of the test are properties of the data that are required by the test to be exact. If some assumption does not hold, the test can still be performed and the p-value computed, but it may be wrong in some sense, e.g. by increasing the risk for false positives (Type I error rate) or by losing statistical power (related to the rate of false negatives, i.e. Type II errors)

The Student's $t$-test and the Welch $t$-test have one assumption in common: they both require that the two populations have normal distribution. But they differ in the assumption about the variances of the population: only the Student's test require them to be equal. This is why the Welch test is sometimes called "unequal variances $t$-test".

To test whether two normally distributed populations have equal variance, you should use other tests such a the $F$ test or Levene's test.

$\endgroup$
  • $\begingroup$ your answer does not address any of my questions. 1. What does it tell us if the means are the same, but variances are different? 2. Can unequal variance test be applied in the example cases I provided? If not, what should I use? $\endgroup$ – Eric Kim Jun 16 at 22:40
  • 1
    $\begingroup$ Have you read it fully? It sounded from your question that you confuse what a test does and its assumptions. I tried to clarify that for you, but I can still see confusion in your new questions. You asked: "How does F-test relate to unequal, or equal variance test? [Welch test]". The answer is: it doesn't. They are 2 entirely different tests, and I explained why. $\endgroup$ – Ous Jun 16 at 23:20
  • $\begingroup$ Again, if variances are somewhat different but you're interested in testing the difference between the means, the Welch test is appropriate. If you want to test the difference between variances, you should go for F test (or better, Levene's test). In the case of two medical procedures having the same mean outcome (let's say a positive one) but different variances, that would mean that both are equally efficient on average, but one produces more consistent results than the other. In that case it may be ethical to favor the consistent one (the one with the lowest variance). $\endgroup$ – Ous Jun 16 at 23:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.