Here I try to reproduce quite old paper from Zimmerman about Mann-Whitney U test and ranked t-test (and also plain t-test) sensitivity to variance differences for assymmetric distributions (I've found copy of this one, but I've got impression that Zimmerman posted this experiment in different journals over decades). Experiment has following idea:
We take sample from exponential distribution
We fit standard scaling on that sample (record it's mean and std)
Then we repeat following procedure 10,000 times:
a) We generate sample A and sample B from exponential distribution
b) We apply standard scaling to sample A and sample B (subtract mean and divide by std we already recorded previously)
c) We don't touch sample A
d) We multiply each element from sample B by 1.05
e) We apply t-test, Mann-Whitney U test and ranked t-test on samples A and B
We have mean=0 by construction both in sample A and sample B. Sample A has std=1, sample B has std=1.05.
- Finally, we calculate % of cases when statistical test gives p-value below 0.05.
And... turns out that this % is about 5% (as expected) for t-test, but 50% (!!!!) for Mann-Whitney U-test and ranked t-test!
Funny thing is that this effect increases when sample size increases. This means that this effect is especially important for commercial A/B testing, when we have samples with thousands items or more.
How this is related to this post? In this post, difference in p-values between Mann-Whitney U-test and ranked t-test is measured and they've found it's small. Also artificial effects were added to empirical distributions, but what about to add artificial variance difference when mean stay the same?
Also it's stated that you should decide what to use based on business need and distribution parameters. Personally, I've struggled with problem of choice between t-test and Mann-Whitney U-test some time ago when we tried to measure average profit of a customer cart. I was amazed by the fact that Mann-Whitney/ranked t-test could be so easily fooled by small difference in variance and this was reason why I resorted to Welch (aka t-test for unequal variances) for my A/B test.
Some people from companies report they doing Mann-Whitney in commercial setting with big group sizes, other resort to t-test. I would be really happy to know right answer for this one, e.g. is it safe to use Mann-Whitney U-test in commercial A/B testing given this problem with unequal variances - because in community different people suggest different things, and quite often it's Mann-Whitney, other times t-test.
So who is right and why? Is it good idea to use ranked t-test/Mann-Whitney for commercial A/B testing (complex distributions with outliers, big sample sizes like 1000 or more)?