3
$\begingroup$

Say I had two distributions and I wanted to determine whether the mechanism behind these two distributions was statistically different.

We know that Student's T-test is pooled, meaning BOTH the distributions must have the same variance. Welch's T-Test is unpooled, meaning these two distributions do not have to be the same.

It seems that Welch's test would be more applicable in the real-world. In almost all cases I can think of, the variance would be different! When does it ever make sense to apply a pooled t-test like Student's t-test?

Should we always check if the variance of the two normal distributions is the same?

$\endgroup$
3
$\begingroup$

Here's one: the pooled test is fine if the sample sizes are equal.

(You should generally not test the assumption of equal variances (numerous posts on site discuss this issue.) If you're not prepared to assume it (and in some situations there's reason to think it should hold), simple don't assume it.

$\endgroup$
2
$\begingroup$

A day-to-day routine:

First of all, how would you know whether in a particular case you may use Student's T test instead of Welch's T test - you simply run a test for the equality of variances, e.g. Levene's test. But this test assumes as null hypothesis the equality of variances and it's rather well known it has a quite weak power, so how you find out if variances are really different? So, it's safer to use Welch's T test as described in article The unequal variance t-test is an underused alternative to Student's t-test and the Mann–Whitney U test by G.D. Ruxton.

The counterexamples:

If you don't have much time, have a glance on this post by Daniel Lakens. Go through all comments and you'll find several entries by Joost de Winter where he points out some strange cases when Student's T test has greater power over Welch's T test, but those examples cover very small sample size. Maybe you would manage to find examples for bigger samples, but he found out that for $n1 = 100$, $n2= 10$, $sd1 = sd2 = 1$ Student's T test has a considerable power advantage of 85% against 78% for Welch's T test. (I believe him, but I haven't verified by myself the validity of his finds).

$\endgroup$
1
$\begingroup$

The more fundamental question is "When is any t-test is a good idea?" since it depends (critically) on having iid normally-distributed samples.

The obvious answer to your question is that it makes sense to apply an unpooled t-test when: the above assumptions are met; you have good reason (such as a scientific model) to believe that the population variances are the same; and the number of samples is very small so that there would be a noteworthy difference between the tests.

I would be surprised if even 1% of t-tests performed in reality, are conducted under these conditions.

$\endgroup$
  • 2
    $\begingroup$ You say that normality is critical; presumably you mean that even moderate deviations from it can have dramatic effects on significance level or power. (otherwise, what does "critically" mean in that sentence?) -- what is the basis for that? $\endgroup$ – Glen_b -Reinstate Monica Feb 25 '16 at 2:49
  • 1
    $\begingroup$ All tests have assumptions which are hardly ever fulfilled perfectly in practice (e.g. exchangability in permutation tests). So it is more relevant to know how robust the tests are with respect to deviations. $\endgroup$ – Michael M Feb 25 '16 at 7:52
  • $\begingroup$ Glen: Normality is critical in exactly the sense you state; small departures can lead to dramatically misleading results. Sadly, I forget the citation, but it was from an article by a famous statistician (maybe Efron?) in a famous journal (probably Annals). Michael: True, but the t-test's assumptions are worse than most. It's a travesty that we teach it by rote, even though it is an antiquated solution to an increasingly irrelevant problem. If statisticians weren't such hide-bound luddites, we could be teaching permutation tests and other empirical resampling methods instead. $\endgroup$ – Timothy Teräväinen Feb 25 '16 at 14:28
1
$\begingroup$

When teaching statistics, there are some advantages of the classic t-test over Welch's modification.

  1. The classic t-test can be computed more easily by paper and pen.

  2. p values from the classic t-test equal the t-test and F-tests from a simple linear regression with a binary covariable. So you can highlight the close relation between the two techniques.

In practice, I don't see any advantage.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy