I want to do an AB test to check if one version does significantly increase revenue. Generically speaking I want to test whether the central tendency (mean) of 2 groups are different from each other on the basis of (unpaired) samples of the 2 groups.
My understanding is, that I could use the following approaches:
- Student-t test, if the variance of both groups are the same and both are normally distributed
- Welch-t test (unequal variance t-test), if the variance of both groups might not be the same but both are still normally distributed
- Mann–Whitney U test (Wilcoxon rank-sum test), if I cannot make any assumptions about the distribution of both groups
But now I read (https://academic.oup.com/beheco/article/17/4/688/215960/The-unequal-variance-t-test-is-an-underused) that I can always use the Welch-t test. The article argues that it is dangerous to make the assumption of equal variance and additionally that
the unequal variance t-test performs as well as, or better than, the Student's t-test in terms of control of both Type I and Type II error rates whenever the underlying distributions are normal.
And if the groups are not normally distributed, than I can just rank the data beforehand:
Thus, Zimmerman and Zumbo (1993) suggest that the unequal variance t-test can effectively replace the Mann–Whitney U test if the data are first ranked before the test is applied.
So the end conclusion is:
If you want to compare the central tendency of 2 populations based on samples of unrelated data, then the unequal variance t-test should always be used in preference to the Student's t-test or Mann–Whitney U test. To use this test, first examine the distributions of the 2 samples graphically. If there is evidence of nonnormality in either or both distributions, then rank the data. Take the ranked or unranked data and perform an unequal variance t-test.
So my question is:
Do you see any drawbacks in always using the Welch-t test instead of the Student-t test or Mann-Whitney test?