Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

It seems like when the assumption of homogeneity of variance is met that the results from a Welch adjusted t-test and a standard t-test are approximately the same. Why not simply always use the Welch adjusted t?

share|improve this question
add comment

6 Answers

up vote 19 down vote accepted

I would like to oppose the other two answers based on a paper (in German) by Kubinger, Rasch and Moder (2009).

They argue, based on "extensive" simulations from distributions either meeting or not meeting the assumptions imposed by a t-test, (normality and homogenity of variance) that the welch-tests performs equally well when the assumptions are met (i.e., basically same probability of committing alpha and beta errors) but outperforms the t-test if the assumptions are not met, especially in terms of power. Therefore, they recommend to always use the welch-test if the sample size exceeds 30.

As a meta-comment: For people interested in statistics (like me and probably most other here) an argument based on data (as mine) should at least count equally as arguments solely based on theoretical grounds (as the others here).


Update:
After thinking about this topic again, I found two further recommendations of which the newer one assists my point. Look at the original papers (which are both, at least for me, freely available) for the argumentations that lead to these recommendations.

The first recommendation comes from Graeme D. Ruxton in 2006: "If you want to compare the central tendency of 2 populations based on samples of unrelated data, then the unequal variance t-test should always be used in preference to the Student's t-test or Mann–Whitney U test."
In:
Ruxton, G.D., 2006. The unequal variance t-test is an underused alternative to Student’s t-test and the Mann–Whitney U test. Behav. Ecol. 17, 688–690.

The second (older) recommendation is from Coombs et al. (1996, p. 148): "In summary, the independent samples t test is generally acceptable in terms of controlling Type I error rates provided there are sufficiently large equal-sized samples, even when the equal population variance assumption is violated. For unequal-sized samples, however, an alternative that does not assume equal population variances is preferable. Use the James second-order test when distributions are either short-tailed symmetric or normal. Promising alternatives include the Wilcox H and Yuen trimmed means tests, which provide broader control of Type I error rates than either the Welch test or the James test and have greater power when data are long-tailed." (emphasis added)
In:
Coombs WT, Algina J, Oltman D. 1996. Univariate and multivariate omnibus hypothesis tests selected to control type I error rates when population variances are not necessarily equal. Rev Educ Res 66:137–79.

share|improve this answer
1  
Meta-response: Good point. But your data might not behave like mine! :-) –  whuber Sep 16 '10 at 23:56
add comment

of course, one could ditch both tests, and start using a Bayesian t-test (Savage-Dickey ratio test), which can account for unequal and unequal variances, and best of all, it allows for a quantification of evidence in favor of the null hypothesis (which means, no more of old "failure to reject" talk)

This test is very simple (and fast) to implement, and there is a paper that clearly explains to readers unfamiliar with Bayesian statistics how to use it, along with an R script. you basically can just insert your data send the commands to the R console:

Wetzels, R., Raaijmakers, J. G. W., Jakab, E., & Wagenmakers, E.-J. (2009). How to Quantify Support For and Against the Null Hypothesis: A Flexible WinBUGS Implementation of a Default Bayesian t-test.

there is also a tutorial for all this, with example data:

http://www.ruudwetzels.com/index.php?src=SDtest

I know this is not a direct response to what was asked, but I thought readers might enjoy having this nice alternative

cheers

share|improve this answer
3  
always these bayesian guys... –  Henrik Jul 27 '10 at 15:45
2  
Another Bayesian alternative to the t-test is Kruschke's BEST (Bayesian estimation supersedes the t test) routine. More info here: indiana.edu/~kruschke/BEST . An online version here: sumsar.net/best_online . –  Rasmus Bååth Jan 8 '13 at 16:19
add comment

Because exact results are preferable to approximations, and avoid odd edge cases where the approximation may lead to a different result than the exact method.

The Welch method isn't a quicker way to do any old t-test, it's a tractable approximation to an otherwise very hard problem: how to construct a t-test under unequal variances. The equal-variance case is well-understood, simple, and exact, and therefore should always be used when possible.

share|improve this answer
3  
I think I tend to agree more with John Tukey -- "Far better an approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise." –  Glen_b Jun 23 '13 at 2:30
add comment

Two reasons I can think of:

  1. Regular Student's T is pretty robust to heteroscedasticity if the sample sizes are equal.

  2. If you believe strongly a priori that the data is homoscedastic, then you lose nothing and might gain a small amount of power by using Studen'ts T instead of Welch's T.

One reason that I would not give is that Student's T is exact and Welch's T isn't. IMHO the exactness of Student's T is academic because it's only exact for normally distributed data, and no real data is exactly normally distributed. I can't think of a single quantity that people actually measure and analyze statistically where the distribution could plausibly have a support of all real numbers. For example, there are only so many atoms in the universe, and some quantities can't be negative. Therefore, when you use any kind of T-test on real data, you're making an approximation anyhow.

share|improve this answer
add comment

The fact that something more complex reduces to something less complex when some assumption is checked is not enough to throw the simpler method away.

share|improve this answer
4  
Especially where students are concerned. –  Matt Parker Jul 20 '10 at 14:51
add comment

I would take the opposite view here. Why bother with the Welch test when the standard unpaired student t test gives you nearly identical results. I studied this issue a while back and I explored a range of scenarios in an attempt to break down the t test and favor the Welch test. To do so I used sample sizes up to 5 times greater for one group vs the other. And, I explored variances up to 25 times greater for one group vs the other. And, it really did not make any material difference. The unpaired t test still generated a range of p values that were nearly identical to the Welch test.

You can see my work at the following link and focus especially on slide 5 and 6.

http://www.slideshare.net/gaetanlion/unpaired-t-test-family

share|improve this answer
    
I'm sorry, what distinction are you making between the large sample formula and the small sample formula? Are you calculating the variances using a population formula in large samples rather than using a sample estimate of the population variance? –  rpierce Sep 17 '10 at 5:53
    
The unpaired student t test has two formulas. The large sample formula is applied to samples with more than 30 observations. The small sample formula is applied to samples with less than 30 observations. The main difference in those formulas is how they calculate the pooled standard error. The small sample formula is much more complicated and counterintuitive. And, in reality it really makes very little difference. I have tested that several times. That's why I think most people have forgotten about this distinction. And, they use most of the time the large sample formula. –  Gaetan Lion Sep 26 '10 at 22:45
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.