Ranked t-test or Mann-Whitney U test for commerical A/B testing

Question

Please help me understand how this blog post relates to this experiment.

Here I try to reproduce quite old paper from Zimmerman about Mann-Whitney U test and ranked t-test (and also plain t-test) sensitivity to variance differences for assymmetric distributions (I've found copy of this one, but I've got impression that Zimmerman posted this experiment in different journals over decades). Experiment has following idea:

1. We take sample from exponential distribution

2. We fit standard scaling on that sample (record it's mean and std)

3. Then we repeat following procedure 10,000 times:

a) We generate sample A and sample B from exponential distribution

b) We apply standard scaling to sample A and sample B (subtract mean and divide by std we already recorded previously)

c) We don't touch sample A

d) We multiply each element from sample B by 1.05

e) We apply t-test, Mann-Whitney U test and ranked t-test on samples A and B

We have mean=0 by construction both in sample A and sample B. Sample A has std=1, sample B has std=1.05.

4. Finally, we calculate % of cases when statistical test gives p-value below 0.05.

And... turns out that this % is about 5% (as expected) for t-test, but 50% (!!!!) for Mann-Whitney U-test and ranked t-test!

Funny thing is that this effect increases when sample size increases. This means that this effect is especially important for commercial A/B testing, when we have samples with thousands items or more.

How this is related to this post? In this post, difference in p-values between Mann-Whitney U-test and ranked t-test is measured and they've found it's small. Also artificial effects were added to empirical distributions, but what about to add artificial variance difference when mean stay the same?

Also it's stated that you should decide what to use based on business need and distribution parameters. Personally, I've struggled with problem of choice between t-test and Mann-Whitney U-test some time ago when we tried to measure average profit of a customer cart. I was amazed by the fact that Mann-Whitney/ranked t-test could be so easily fooled by small difference in variance and this was reason why I resorted to Welch (aka t-test for unequal variances) for my A/B test.

Some people from companies report they doing Mann-Whitney in commercial setting with big group sizes, other resort to t-test. I would be really happy to know right answer for this one, e.g. is it safe to use Mann-Whitney U-test in commercial A/B testing given this problem with unequal variances - because in community different people suggest different things, and quite often it's Mann-Whitney, other times t-test.

So who is right and why? Is it good idea to use ranked t-test/Mann-Whitney for commercial A/B testing (complex distributions with outliers, big sample sizes like 1000 or more)?

Answer 1

My own thoughts on this:

Before deciding what to choose, Mann-Whitney U test or t-test, or other test, you can do or simulate A/A test to obtain samples to look at. Usually in A/B tests you compare some existing option with new one (or many new ones), and in this scenario you should have at least some historical data after you've started to prepare test infrastructure (you need to record samples somehow after all). For my cart profit example, you can look at recorded clickstream data, simulate desired (or just plain random if you can't really simulate that) customer selection and observe difference in variance between groups. You also should do real A/A test while doing your A/B test to catch any odd stuff in your procedure/environment anyway.

Next, if you plan to include thousands of samples into group and observe variance difference, say of 1% or more, then doing Mann-Whitney or ranked t-test maybe not that good idea after all. If your sample sizes will be small, like 100 or something like that, then it's maybe worth trying.

On deeper level though, reason of this behavior is simply because when location shift (read: variance equality assuming same underlying distribution in both groups and just maybe different medians) assumption doesn't hold, you can't interpret such test result as "medians are different or not" statement. Instead it says just that distributions are different (meaning underlying distribution is different or, in my example, that variance is different). And question is, does it really what you want to check with such test and is this how you interpret it?

On even more deeper level, a look at your historical data (which I suggested above) may hint you to use some third option as a test provided you have enough knowledge in applied statistics.