47
$\begingroup$

It seems like when the assumption of homogeneity of variance is met that the results from a Welch adjusted t-test and a standard t-test are approximately the same. Why not simply always use the Welch adjusted t?

$\endgroup$
33
$\begingroup$

I would like to oppose the other two answers based on a paper (in German) by Kubinger, Rasch and Moder (2009).

They argue, based on "extensive" simulations from distributions either meeting or not meeting the assumptions imposed by a t-test, (normality and homogenity of variance) that the welch-tests performs equally well when the assumptions are met (i.e., basically same probability of committing alpha and beta errors) but outperforms the t-test if the assumptions are not met, especially in terms of power. Therefore, they recommend to always use the welch-test if the sample size exceeds 30.

As a meta-comment: For people interested in statistics (like me and probably most other here) an argument based on data (as mine) should at least count equally as arguments solely based on theoretical grounds (as the others here).


Update:
After thinking about this topic again, I found two further recommendations of which the newer one assists my point. Look at the original papers (which are both, at least for me, freely available) for the argumentations that lead to these recommendations.

The first recommendation comes from Graeme D. Ruxton in 2006: "If you want to compare the central tendency of 2 populations based on samples of unrelated data, then the unequal variance t-test should always be used in preference to the Student's t-test or Mann–Whitney U test."
In:
Ruxton, G.D., 2006. The unequal variance t-test is an underused alternative to Student’s t-test and the Mann–Whitney U test. Behav. Ecol. 17, 688–690.

The second (older) recommendation is from Coombs et al. (1996, p. 148): "In summary, the independent samples t test is generally acceptable in terms of controlling Type I error rates provided there are sufficiently large equal-sized samples, even when the equal population variance assumption is violated. For unequal-sized samples, however, an alternative that does not assume equal population variances is preferable. Use the James second-order test when distributions are either short-tailed symmetric or normal. Promising alternatives include the Wilcox H and Yuen trimmed means tests, which provide broader control of Type I error rates than either the Welch test or the James test and have greater power when data are long-tailed." (emphasis added)
In:
Coombs WT, Algina J, Oltman D. 1996. Univariate and multivariate omnibus hypothesis tests selected to control type I error rates when population variances are not necessarily equal. Rev Educ Res 66:137–79.

$\endgroup$
  • 3
    $\begingroup$ Meta-response: Good point. But your data might not behave like mine! :-) $\endgroup$ – whuber Sep 16 '10 at 23:56
  • $\begingroup$ Henrik, would you mind if I edit the answer to: (1) change the terminology by calling the tests Student's t-test and Welch's t-test (as I've found in most of the literatura); (2) include another paper that suggests it in the discussion: rips-irsp.com/article/10.5334/irsp.82 (it emphasizes the bias that happens when you pick the tests based in the Levene's test of homogeneity). $\endgroup$ – Bruno Nov 12 '18 at 1:28
13
$\begingroup$

of course, one could ditch both tests, and start using a Bayesian t-test (Savage-Dickey ratio test), which can account for unequal and unequal variances, and best of all, it allows for a quantification of evidence in favor of the null hypothesis (which means, no more of old "failure to reject" talk)

This test is very simple (and fast) to implement, and there is a paper that clearly explains to readers unfamiliar with Bayesian statistics how to use it, along with an R script. you basically can just insert your data send the commands to the R console:

Wetzels, R., Raaijmakers, J. G. W., Jakab, E., & Wagenmakers, E.-J. (2009). How to Quantify Support For and Against the Null Hypothesis: A Flexible WinBUGS Implementation of a Default Bayesian t-test.

there is also a tutorial for all this, with example data:

http://www.ruudwetzels.com/index.php?src=SDtest

I know this is not a direct response to what was asked, but I thought readers might enjoy having this nice alternative

cheers

$\endgroup$
  • 8
    $\begingroup$ always these bayesian guys... $\endgroup$ – Henrik Jul 27 '10 at 15:45
  • 3
    $\begingroup$ Another Bayesian alternative to the t-test is Kruschke's BEST (Bayesian estimation supersedes the t test) routine. More info here: indiana.edu/~kruschke/BEST . An online version here: sumsar.net/best_online . $\endgroup$ – Rasmus Bååth Jan 8 '13 at 16:19
7
$\begingroup$

Because exact results are preferable to approximations, and avoid odd edge cases where the approximation may lead to a different result than the exact method.

The Welch method isn't a quicker way to do any old t-test, it's a tractable approximation to an otherwise very hard problem: how to construct a t-test under unequal variances. The equal-variance case is well-understood, simple, and exact, and therefore should always be used when possible.

$\endgroup$
  • 6
    $\begingroup$ I think I tend to agree more with John Tukey -- "Far better an approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise." $\endgroup$ – Glen_b Jun 23 '13 at 2:30
  • 4
    $\begingroup$ The equal-variance (Student) t-test itself is merely an (ill-understood) approximation when the population sample variances are unequal. Therefore, unless it is known that the population variances are equal, it is better to use an approximation to the correct sampling distribution (the Welch-Satterthwaite) than to use a perfectly accurate distribution that does not apply to the data model. $\endgroup$ – whuber Sep 24 '14 at 21:33
4
$\begingroup$

Two reasons I can think of:

  1. Regular Student's T is pretty robust to heteroscedasticity if the sample sizes are equal.

  2. If you believe strongly a priori that the data is homoscedastic, then you lose nothing and might gain a small amount of power by using Studen'ts T instead of Welch's T.

One reason that I would not give is that Student's T is exact and Welch's T isn't. IMHO the exactness of Student's T is academic because it's only exact for normally distributed data, and no real data is exactly normally distributed. I can't think of a single quantity that people actually measure and analyze statistically where the distribution could plausibly have a support of all real numbers. For example, there are only so many atoms in the universe, and some quantities can't be negative. Therefore, when you use any kind of T-test on real data, you're making an approximation anyhow.

$\endgroup$
  • 2
    $\begingroup$ (1) is incorrect when the underlying population variances are greatly different. As an extreme case--to see why this is so--consider what happens when one population has no variance at all. The Student t would in effect be comparing data from the other population to a constant, but it would think it has twice as many degrees of freedom. The error it makes would be comparable to just using a Z test. $\endgroup$ – whuber Sep 24 '14 at 21:37
  • $\begingroup$ While this is true @whuber it's only for very extreme cases. I was just looking at a 1e6:1 variance difference and p ≈ .053. So it can happen but I'd still argue it's pretty robust with equal N. $\endgroup$ – John Jun 26 '17 at 12:11
  • $\begingroup$ @John A 1e6:1 variance ratio is huge, regardless of what the $n_i$ might be. The Student's t is likely to be misleading in that case. $\endgroup$ – whuber Jun 26 '17 at 14:32
  • $\begingroup$ @whuber, I'm just suggesting that while your comment above is technically true, Welch correction isn't the solution to the problem you pose as an example and it's not even really very critical of the robustness of the test in terms of alpha rates (which is what (1) usually means). As you suggest, when (extreme) unequal variance is an issue you've got other problems, but that's really a different topic. $\endgroup$ – John Jun 27 '17 at 14:08
3
$\begingroup$

The fact that something more complex reduces to something less complex when some assumption is checked is not enough to throw the simpler method away.

$\endgroup$
  • 4
    $\begingroup$ Especially where students are concerned. $\endgroup$ – Matt Parker Jul 20 '10 at 14:51
2
$\begingroup$

I would take the opposite view here. Why bother with the Welch test when the standard unpaired student t test gives you nearly identical results. I studied this issue a while back and I explored a range of scenarios in an attempt to break down the t test and favor the Welch test. To do so I used sample sizes up to 5 times greater for one group vs the other. And, I explored variances up to 25 times greater for one group vs the other. And, it really did not make any material difference. The unpaired t test still generated a range of p values that were nearly identical to the Welch test.

You can see my work at the following link and focus especially on slide 5 and 6.

http://www.slideshare.net/gaetanlion/unpaired-t-test-family

$\endgroup$
  • $\begingroup$ I'm sorry, what distinction are you making between the large sample formula and the small sample formula? Are you calculating the variances using a population formula in large samples rather than using a sample estimate of the population variance? $\endgroup$ – russellpierce Sep 17 '10 at 5:53
  • $\begingroup$ The unpaired student t test has two formulas. The large sample formula is applied to samples with more than 30 observations. The small sample formula is applied to samples with less than 30 observations. The main difference in those formulas is how they calculate the pooled standard error. The small sample formula is much more complicated and counterintuitive. And, in reality it really makes very little difference. I have tested that several times. That's why I think most people have forgotten about this distinction. And, they use most of the time the large sample formula. $\endgroup$ – Sympa Sep 26 '10 at 22:45
0
$\begingroup$

It's true that the frequentist properties of the Welch corrected test are better than the ordinary Student's T, at least for errors. I agree that that alone is a pretty good argument for the Welch test. However, I'm usually reluctant to recommend the Welch correction because it's use is often deceptive. Which is, admittedly not a critique of the test itself.

The reason I don't recommend the Welch correction is that it doesn't just change the degrees of freedom and subsequent theoretical distribution from which the p-value is drawn. It makes the test non-parametric. To perform a Welch corrected t-test one still pools variance as if equal variance can be assumed but then changes the final testing procedure implying either that equal variance cannot be assumed, or that you only care about the sample variances. This makes it a non-parametric test because the pooled variance is considered non-representative of the population and you conceded that you're just testing your observed values.

In and of itself there's nothing particularly wrong with that. However, I find it deceptive because a) typically it's not reported with enough specificity; and b) the people who use it tend to think about it interchangeably with a t-test. The only way I ever know that it has been done in published papers is when I see an odd DF for the t-distribution. That was also the only way Rexton (referenced in the Henrik answer) could tell in review. Unfortunately, the non-parametric nature of the Welch corrected test occurs whether the degrees of freedom have changed or not (i.e. even if the sample variances are equal). But this reporting issue is symptomatic of the fact that most people who use the Welch correction don't recognize this change to the test has occurred. I just interviewed a few colleagues and they admitted they had never even thought of it.

Therefore, because of this, I believe that if you're going to recommend a non-parametric test don't use one that often appears parametric or at least be very clear about what you're doing. The official name of the test should be Non-Parametric Welch Corrected T-test. If people reported it that way I'd be much happier with Henrik's recommendation.

$\endgroup$
  • $\begingroup$ I couldn't find any support in your answer for why the Welch test might be "deceptive." Could you explain the basis for that? $\endgroup$ – whuber Jun 27 '17 at 20:00
  • $\begingroup$ Perhaps my edits clarified things @whuber. I should have been clear that it's not guaranteed to be deceptive but often is to both the user of the test and reader of the test results. $\endgroup$ – John Jun 27 '17 at 20:06
  • 1
    $\begingroup$ Thank you. Apart from the reporting issue--which it would be unfair to characterize as a fault of the test!--this seems to come down to some sort of objection on your part that the Welch test is non-parametric. What could possibly be the matter with that? Ceteris paribus, that has to be considered an advantage, not a problem. $\endgroup$ – whuber Jun 27 '17 at 21:32
  • 1
    $\begingroup$ It's a distinction that's generally not made clear. I admit in the answer that it's not in and of itself a problem but most people tend to treat it parametrically, which is an error. I don't think here is the place to have the discussion about benefits or costs of non-parametric testing. Also, it wasn't mentioned in the thread and it may be an issue for many people. As an aside, two of our intro stats class teach it in parallel with the Student t-test, and promote it, but have a whole separate section on non-parametric tests. $\endgroup$ – John Jun 28 '17 at 13:37
  • $\begingroup$ Can you clarify what you mean by "makes the test nonparametric"? $\endgroup$ – Glen_b Oct 18 '17 at 9:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.