Mann Whitney U versus bootstrapped sampling distribution of difference between means

Question

Context: I want to run an AB test to measure the effect of an algorithmic change and whether the effect is statistically significant. My north star metric, the CPC (cost per click), is not normally distributed. It looks much more like a mixture of 2 Gaussian random variables. There is really no transformation that I know off that can make it normally distributed.

I want to estimate the sample size required to detect an effect size of 5% of the mean with a significance level (alpha) of 5% and a power (1-beta) of 80%.

Given that the CPC does not follow any known statistical distribution law I have been looking into using a non parametric data driven method to estimate the sample size required. In this procedure, once a specific test has been agreed upon and given 2 samples of a specific size I need to compute the power of the test.

Looking on the internet, the only procedure that I could find seems to rely on the Mann Whitney U test and would look like that:

Given 2 samples A and B (both of size 12000):

1. Draw 10000 new samples of size 12000 each with replacement from A
2. Draw 10000 new samples of size 12000 each with replacement from B
3. For every 10000 new sample pairs (A', B') apply the Mann Whitney U test
4. Return power == (number of times p-value < alpha) / 10000

But now I am thinking of using a different method that would rely on law of large numbers and the bootstrap:

Given 2 samples A and B (both of size 12000):

1. Draw 10000 new samples of size 12000 with replacement from A
2. Compute the means of these 10000 samples
3. Draw 10000 new samples of size 12000 with replacement from B
4. Compute the means of these 10000 other samples
5. Compute the difference between the 2 groups of means (thanks to the central limit theorem we know that the result will be normally distributed)
6. Among the 10000 trials we want to compute the number of times the difference is too big for the null hypothesis to be true. I see 3 options:
   • Compute the standard error of the 10000 A' samples and compute the number of absolute mean differences that greater than 1.96 * standard error.
   • Compute the standard deviation of the difference between means and the number of absolute mean differences that greater than 1.96 * standard deviation.
   • Compute the number of absolute mean differences that are lower than the 0.025 quantile of the distribution or greater than the 0.975 quantile.

Nota bene:

• The 3 concluding options should be equivalent
• Options 6.a and 6.b are equivalent because, thanks to the variance sum law, the standard error of the mean is the standard deviation of the sampling distribution of the mean.

Now from from what I understand the procedure with the Mann Whitney U test would only allow me to detect a difference in medians. Given that the difference in either the mean or the median is normally distributed, the second procedure seems applicable in a more diverse set of situations (I think I can also apply it for some other quantiles).

I ran then 2 procedure computing a difference in medians and I get almost the same result as when using the Mann Whitney U test.

Question It seems weird to me that I did not find any material online mentioning such an approach. Apart from any computational efficiency considerations, do you think that this procedure is valid? If not, why?

Answer 1

The Wilcoxon-Mann-Whitney 2-sample rank-sum test is a special case of the proportional odds semiparametric ordinal logistic model. The R Hmisc package contains two functions for power/sample size for the Wilcoxon: popower, posamsize. You must provide an estimate of the entire frequency distribution of Y to use these functions. Once you do that you have a power calculation that is tailored to your distribution, whether it's heavy tailed, asymmetric, multimodal, etc. A detailed example is in the nonparametric statistics chapter of BBR. The example also shows you how to translate odds ratios to differences in means or other measures.