4
$\begingroup$

I want to do an AB test to check if one version does significantly increase revenue. Generically speaking I want to test whether the central tendency (mean) of 2 groups are different from each other on the basis of (unpaired) samples of the 2 groups.

My understanding is, that I could use the following approaches:

  • Student-t test, if the variance of both groups are the same and both are normally distributed
  • Welch-t test (unequal variance t-test), if the variance of both groups might not be the same but both are still normally distributed
  • Mann–Whitney U test (Wilcoxon rank-sum test), if I cannot make any assumptions about the distribution of both groups

But now I read (https://academic.oup.com/beheco/article/17/4/688/215960/The-unequal-variance-t-test-is-an-underused) that I can always use the Welch-t test. The article argues that it is dangerous to make the assumption of equal variance and additionally that

the unequal variance t-test performs as well as, or better than, the Student's t-test in terms of control of both Type I and Type II error rates whenever the underlying distributions are normal.

And if the groups are not normally distributed, than I can just rank the data beforehand:

Thus, Zimmerman and Zumbo (1993) suggest that the unequal variance t-test can effectively replace the Mann–Whitney U test if the data are first ranked before the test is applied.

So the end conclusion is:

If you want to compare the central tendency of 2 populations based on samples of unrelated data, then the unequal variance t-test should always be used in preference to the Student's t-test or Mann–Whitney U test. To use this test, first examine the distributions of the 2 samples graphically. If there is evidence of nonnormality in either or both distributions, then rank the data. Take the ranked or unranked data and perform an unequal variance t-test.

So my question is:

Do you see any drawbacks in always using the Welch-t test instead of the Student-t test or Mann-Whitney test?

$\endgroup$
  • $\begingroup$ "unequal variance t-test can effectively replace the Mann–Whitney U test if the data are first ranked before the test" - what do they mean by this? apply Welch-corrected t-test at rankings? I doubt that it is a correct procedure. Or did I understood this wrong? $\endgroup$ – German Demidov Nov 13 '17 at 16:49
  • $\begingroup$ first of all, Mann-Whitney is the comparison of medians while both forms of t-test are comparisons of means. For equal variances case t-test with and without Welch correction will show ~ the same result. Of course t-test without Welch correction is not correct of unequal variances case. IDK why the author of the paper you refer to bothered to write a paper about this. MU test is not correct for unequal variances too BTW, but t-test and MU should undergo only limited comparison for RVs distributed in a way that mean and median is equal. $\endgroup$ – German Demidov Nov 13 '17 at 16:54
  • $\begingroup$ Welch correction of unequal variances is also non-exact (but useful in a lot of cases) solution (en.wikipedia.org/wiki/Behrens–Fisher_problem) $\endgroup$ – German Demidov Nov 13 '17 at 17:00
  • $\begingroup$ Quite useful explanation: daniellakens.blogspot.com.es/2015/01/… $\endgroup$ – German Demidov Nov 13 '17 at 19:01
7
$\begingroup$

There have been a number of papers which examine this issue. Most of them come to the conclusion that Welch's version of the t-test can be safely used in most circumstances.

The only situation in which the test seems to have undesirable performance is in very small sample sizes.

Here are some quotes from two papers which examine t-test performance with small sample sizes:

The t-test with the unequal variances option (i.e., the Welch test) was generally not preferred either. Only in the case of unequal variances combined with unequal sample sizes, where the small sample was drawn from the small variance population, did this approach provide a power advantage compared to the regular ttest. In the other cases, a substantial amount of statistical power was lost compared to the regular t-test. The power loss of the Welch test can be explained by its lower degrees of freedom determined from the Welch-Satterthwaite equation.$^1$

Results suggest that the Welch t test is indeed inflated, according to Bradley's (1978) fairly stringent criterion, when sample sizes are unequal – even when assumptions for the t test are met in the population. The inflation rate seems to be dependent more on the size of the smaller group than on the total sample size, but sample size ratio does seem to play a small role$^2$

If you read through those papers though, you'll see that it's really only in the specific case with very small sample sizes (in particular, when the smaller of the two groups is very small) that it's much of an issue. "Small" meaning the effects are really only troublesome when a group contains around 5 subjects or less as posited by both papers, but take a closer look at the references for a more thorough discussion. In that case, you might (obviously) suggest collecting more data. But this can of course be an issue with prohibitively expensive experiments.

Otherwise Welch's is probably fine.

$^1$ : Using the Student’s t-test with extremely small sample sizes, J.C.F. de Winter 2013

$^2$ : Type I Error Inflation of the Separate-Variances Welch t test with Very Small Sample Sizes when Assumptions Are Met, Albert K. Adusah and Gordon P. Brooks 2011

$\endgroup$
  • $\begingroup$ What is "small"? 10? 100? $\endgroup$ – xan Nov 13 '17 at 20:38
  • $\begingroup$ I agree with the idea that if your samples are extremely small, any conclusions are not meaningful anyway. $\endgroup$ – qwr Apr 5 '18 at 21:25
4
$\begingroup$

Do you see any drawbacks in always using the Welch-t test instead of the Student-t test or Mann-Whitney test?

Welch t-tests are inherently made for unpaired samples. If you have paired samples because you measured the same subjects twice, you should use that information. Paired sample t-tests are more powerful.

This one is the most common exception, paired data can occur in most if not all fields of research. There are also specific variants of t-tests that correct for specific kinds of pseudo-replication in specific contexts. For example, there are corrected resampled t-tests for data that comes from repeated cross validation. Those variants don't usually integrate with Welch's approach either.

$\endgroup$
  • $\begingroup$ I have unpaired samples. I will make this clear in my question. $\endgroup$ – Pascal Klein Nov 13 '17 at 15:14
  • $\begingroup$ Or you could leave it like this since the question is framed in general and not for a specific data-set. Paired samples are the most relevant exception to using Welch tests, they happen in most if not all fields of research. $\endgroup$ – David Ernst Nov 13 '17 at 15:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.