It seems like when the assumption of homogeneity of variance is met that the results from a Welch adjusted t-test and a standard t-test are approximately the same. Why not simply always use the Welch adjusted t?
When conducting a t-test why would one prefer to assume (or test for) equal variances rather than always use a Welch approximation of the df?
I would like to oppose the other two answers based on a paper (in German) by Kubinger, Rasch and Moder (2009).
They argue, based on "extensive" simulations from distributions either meeting or not meeting the assumptions imposed by a t-test, (normality and homogenity of variance) that the welch-tests performs equally well when the assumptions are met (i.e., basically same probability of committing alpha and beta errors) but outperforms the t-test if the assumptions are not met, especially in terms of power. Therefore, they recommend to always use the welch-test if the sample size exceeds 30.
As a meta-comment: For people interested in statistics (like me and probably most other here) an argument based on data (as mine) should at least count equally as arguments solely based on theoretical grounds (as the others here).
After thinking about this topic again, I found two further recommendations of which the newer one assists my point. Look at the original papers (which are both, at least for me, freely available) for the argumentations that lead to these recommendations.
The first recommendation comes from Graeme D. Ruxton in 2006: "If you want to compare the central tendency of 2 populations based on samples of unrelated data, then the unequal variance t-test should always be used in preference to the Student's t-test or Mann–Whitney U test."
Ruxton, G.D., 2006. The unequal variance t-test is an underused alternative to Student’s t-test and the Mann–Whitney U test. Behav. Ecol. 17, 688–690.
The second (older) recommendation is from Coombs et al. (1996, p. 148): "In summary, the independent samples t test is generally acceptable in terms of controlling Type I error rates provided there are sufficiently large equal-sized samples, even when the equal population variance assumption is violated. For unequal-sized samples, however, an alternative that does not assume equal population variances is preferable. Use the James second-order test when distributions are either short-tailed symmetric or normal. Promising alternatives include the Wilcox H and Yuen trimmed means tests, which provide broader control of Type I error rates than either the Welch test or the James test and have greater power when data are long-tailed." (emphasis added)
Coombs WT, Algina J, Oltman D. 1996. Univariate and multivariate omnibus hypothesis tests selected to control type I error rates when population variances are not necessarily equal. Rev Educ Res 66:137–79.
3$\begingroup$ Meta-response: Good point. But your data might not behave like mine! :-) $\endgroup$ Sep 16, 2010 at 23:56
2$\begingroup$ Henrik, would you mind if I edit the answer to: (1) change the terminology by calling the tests Student's t-test and Welch's t-test (as I've found in most of the literatura); (2) include another paper that suggests it in the discussion: rips-irsp.com/article/10.5334/irsp.82 (it emphasizes the bias that happens when you pick the tests based in the Levene's test of homogeneity). $\endgroup$– BrunoNov 12, 2018 at 1:28
Of course, one could ditch both tests, and start using a Bayesian t-test (Savage-Dickey ratio test), which can account for equal and unequal variances, and best of all, it allows for a quantification of evidence in favor of the null hypothesis (which means, no more of old "failure to reject" talk)
This test is very simple (and fast) to implement, and there is a paper that clearly explains to readers unfamiliar with Bayesian statistics how to use it, along with an R script. You basically can just insert your data send the commands to the R console:
Wetzels, R., Raaijmakers, J. G. W., Jakab, E., & Wagenmakers, E.-J. (2009). How to Quantify Support For and Against the Null Hypothesis: A Flexible WinBUGS Implementation of a Default Bayesian t-test.
there is also a tutorial for all this, with example data:
I know this is not a direct response to what was asked, but I thought readers might enjoy having this nice alternative
9$\begingroup$ always these bayesian guys... $\endgroup$– HenrikJul 27, 2010 at 15:45
4$\begingroup$ Another Bayesian alternative to the t-test is Kruschke's BEST (Bayesian estimation supersedes the t test) routine. More info here: indiana.edu/~kruschke/BEST . An online version here: sumsar.net/best_online . $\endgroup$ Jan 8, 2013 at 16:19
Because exact results are preferable to approximations, and avoid odd edge cases where the approximation may lead to a different result than the exact method.
The Welch method isn't a quicker way to do any old t-test, it's a tractable approximation to an otherwise very hard problem: how to construct a t-test under unequal variances. The equal-variance case is well-understood, simple, and exact, and therefore should always be used when possible.
9$\begingroup$ I think I tend to agree more with John Tukey -- "Far better an approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise." $\endgroup$– Glen_bJun 23, 2013 at 2:30
5$\begingroup$ The equal-variance (Student) t-test itself is merely an (ill-understood) approximation when the population sample variances are unequal. Therefore, unless it is known that the population variances are equal, it is better to use an approximation to the correct sampling distribution (the Welch-Satterthwaite) than to use a perfectly accurate distribution that does not apply to the data model. $\endgroup$ Sep 24, 2014 at 21:33
$\begingroup$ If you really need a strict control over Type I error rate you could do worse than Behrens & Fisher's original test. It is exact in the sense that e.g. Fisher's Exact Test is exact: Type I errors are guaranteed not to exceed the nominal rate, whatever the true values of the nuisance parameters. (And its rejection regions are sensible ones.) $\endgroup$ Dec 6, 2022 at 11:47
Two reasons I can think of:
Regular Student's T is pretty robust to heteroscedasticity if the sample sizes are equal.
If you believe strongly a priori that the data is homoscedastic, then you lose nothing and might gain a small amount of power by using Studen'ts T instead of Welch's T.
One reason that I would not give is that Student's T is exact and Welch's T isn't. IMHO the exactness of Student's T is academic because it's only exact for normally distributed data, and no real data is exactly normally distributed. I can't think of a single quantity that people actually measure and analyze statistically where the distribution could plausibly have a support of all real numbers. For example, there are only so many atoms in the universe, and some quantities can't be negative. Therefore, when you use any kind of T-test on real data, you're making an approximation anyhow.
3$\begingroup$ (1) is incorrect when the underlying population variances are greatly different. As an extreme case--to see why this is so--consider what happens when one population has no variance at all. The Student t would in effect be comparing data from the other population to a constant, but it would think it has twice as many degrees of freedom. The error it makes would be comparable to just using a Z test. $\endgroup$ Sep 24, 2014 at 21:37
$\begingroup$ While this is true @whuber it's only for very extreme cases. I was just looking at a 1e6:1 variance difference and p ≈ .053. So it can happen but I'd still argue it's pretty robust with equal N. $\endgroup$ Jun 26, 2017 at 12:11
$\begingroup$ @John A 1e6:1 variance ratio is huge, regardless of what the $n_i$ might be. The Student's t is likely to be misleading in that case. $\endgroup$ Jun 26, 2017 at 14:32
$\begingroup$ @whuber, I'm just suggesting that while your comment above is technically true, Welch correction isn't the solution to the problem you pose as an example and it's not even really very critical of the robustness of the test in terms of alpha rates (which is what (1) usually means). As you suggest, when (extreme) unequal variance is an issue you've got other problems, but that's really a different topic. $\endgroup$ Jun 27, 2017 at 14:08
The fact that something more complex reduces to something less complex when some assumption is checked is not enough to throw the simpler method away.
4$\begingroup$ Especially where students are concerned. $\endgroup$ Jul 20, 2010 at 14:51
I would take the opposite view here. Why bother with the Welch test when the standard unpaired student t test gives you nearly identical results. I studied this issue a while back and I explored a range of scenarios in an attempt to break down the t test and favor the Welch test. To do so I used sample sizes up to 5 times greater for one group vs the other. And, I explored variances up to 25 times greater for one group vs the other. And, it really did not make any material difference. The unpaired t test still generated a range of p values that were nearly identical to the Welch test.
You can see my work at the following link and focus especially on slide 5 and 6.
$\begingroup$ I'm sorry, what distinction are you making between the large sample formula and the small sample formula? Are you calculating the variances using a population formula in large samples rather than using a sample estimate of the population variance? $\endgroup$ Sep 17, 2010 at 5:53
$\begingroup$ The unpaired student t test has two formulas. The large sample formula is applied to samples with more than 30 observations. The small sample formula is applied to samples with less than 30 observations. The main difference in those formulas is how they calculate the pooled standard error. The small sample formula is much more complicated and counterintuitive. And, in reality it really makes very little difference. I have tested that several times. That's why I think most people have forgotten about this distinction. And, they use most of the time the large sample formula. $\endgroup$– SympaSep 26, 2010 at 22:45
With the assumption of equal variance, one can derive the non asymptotic distribution of t statistics. But when the assumption is violated, two variance terms cannot be cancelled, and we cannot simplify the distribution of statistics to a fixed one. Thus test can't be done.
Welch t test is an approximation which is robust and give an approximate degree of freedom. But it is not "exact", which means its type one error is not exactly what you want theoretically.
From my perspective, even when homogeneity test doesn't reject "equal variance", there is still risk to use t test assuming same variance. Because the true variance difference may be small/not significant, but not zero. We need a test without relying on "same variance", rather than use homogeneity test for "same variance".
https://arxiv.org/abs/2210.16473. Here is my new paper, hope it can help with your question. It can derive "exact" or non-asymptotic t statistics when variances of two groups are different, and it reaches the maximal degree of freedom that an "exact test" can allow. In small sample cases, it significantly outperforms Welch's t-test, in the sense of type one error. ($\mu_1=\mu_2=0$,$\sigma_1=1,\sigma_2=2,n_1=5,n_2=50$)
Its idea is: paired t test can always give an exact test, even when variances are unequal. But it loses some information when n1<n2, because it can only use n1 data points from n2 samples. This paper uses an orthogonal matrix to project the longer vector (n2) into a length (n1) vector, then we can use paired t test, with enough/compressed information.
Go here for instruction of use, a package developed by me: https://github.com/hobbitish1028/Te_test
$\begingroup$ That's ingenious, but any inferences you make from i.i.d. samples oughtn't to depend on their ordering. The same can be said, of course, of Scheffé's solution of the Behrens-Fisher problem; he later wrote that he developed it "before I had much consulting experience, and since then I have never recommended the solution in practice" (Scheffé (1970), J. Am. Stat. Assoc., 65, 332, "Practical Solutions of the Behrens-Fisher Problem"). $\endgroup$ Nov 7, 2022 at 9:10
It's true that the frequentist properties of the Welch corrected test are better than the ordinary Student's T, at least for errors. I agree that that alone is a pretty good argument for the Welch test. However, I'm usually reluctant to recommend the Welch correction because it's use is often deceptive. Which is, admittedly not a critique of the test itself.
The reason I don't recommend the Welch correction is that it doesn't just change the degrees of freedom and subsequent theoretical distribution from which the p-value is drawn. It makes the test non-parametric. To perform a Welch corrected t-test one still pools variance as if equal variance can be assumed but then changes the final testing procedure implying either that equal variance cannot be assumed, or that you only care about the sample variances. This makes it a non-parametric test because the pooled variance is considered non-representative of the population and you conceded that you're just testing your observed values.
In and of itself there's nothing particularly wrong with that. However, I find it deceptive because a) typically it's not reported with enough specificity; and b) the people who use it tend to think about it interchangeably with a t-test. The only way I ever know that it has been done in published papers is when I see an odd DF for the t-distribution. That was also the only way Rexton (referenced in the Henrik answer) could tell in review. Unfortunately, the non-parametric nature of the Welch corrected test occurs whether the degrees of freedom have changed or not (i.e. even if the sample variances are equal). But this reporting issue is symptomatic of the fact that most people who use the Welch correction don't recognize this change to the test has occurred. I just interviewed a few colleagues and they admitted they had never even thought of it.
Therefore, because of this, I believe that if you're going to recommend a non-parametric test don't use one that often appears parametric or at least be very clear about what you're doing. The official name of the test should be Non-Parametric Welch Corrected T-test. If people reported it that way I'd be much happier with Henrik's recommendation.
1$\begingroup$ I couldn't find any support in your answer for why the Welch test might be "deceptive." Could you explain the basis for that? $\endgroup$ Jun 27, 2017 at 20:00
$\begingroup$ Perhaps my edits clarified things @whuber. I should have been clear that it's not guaranteed to be deceptive but often is to both the user of the test and reader of the test results. $\endgroup$ Jun 27, 2017 at 20:06
2$\begingroup$ Thank you. Apart from the reporting issue--which it would be unfair to characterize as a fault of the test!--this seems to come down to some sort of objection on your part that the Welch test is non-parametric. What could possibly be the matter with that? Ceteris paribus, that has to be considered an advantage, not a problem. $\endgroup$ Jun 27, 2017 at 21:32
1$\begingroup$ It's a distinction that's generally not made clear. I admit in the answer that it's not in and of itself a problem but most people tend to treat it parametrically, which is an error. I don't think here is the place to have the discussion about benefits or costs of non-parametric testing. Also, it wasn't mentioned in the thread and it may be an issue for many people. As an aside, two of our intro stats class teach it in parallel with the Student t-test, and promote it, but have a whole separate section on non-parametric tests. $\endgroup$ Jun 28, 2017 at 13:37
1$\begingroup$ Can you clarify what you mean by "makes the test nonparametric"? $\endgroup$– Glen_bOct 18, 2017 at 9:11