3
$\begingroup$

I am looking for solutions around non-parametric hypothesis testing on revenue metrics in business settings. Currently I face with difficulties in finding the right way to make a Mann-Whitney U test results' effect size easily interpretable to the business. The intention would be to show the uplift of revenue for significant a/b tests. Translating the common language effect size or rank correlation to them is quite challenging and generally their interest is mainly towards the magnitude, the revenue uplift estimates.

Can you advise me any techniques?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
5
$\begingroup$

The common language effect size statistic † is an appropriate effect size statistic to pair with a Wilcoxon-Mann-Whitney (WMW) test. It is simply the probability of an observation in one group being larger than an observation in another group. That should be pretty easy for your audience to understand, but that doesn't mean that they care about such a statistic.

If you are really interested in the magnitude of the difference in revenue, WMW or the common language effect size statistic may not be what you're really interested in. You might be more interested in comparing the means of the two groups or the medians of the two groups. Or, honestly, if your audience just wants the projected difference in revenue, this value may be the only thing of interest, and a statistical test and effect size statistic may not be meaningful to them anyway.

† Also called Vargha and Delaney's A.

Improve this answer
$\endgroup$
5
  • $\begingroup$ This concordance probability Sal mentioned is an excellent summary. If you want to summarize MWM using odds ratios, means, exceedance probabilities, and quantiles, see Section 7.6 of BBR. $\endgroup$
    – Frank Harrell
    Commented Jan 26, 2020 at 15:54
  • $\begingroup$ Thanks both; very valid points. From the business side, they are used to conducting t-test (on right skewed distribution) and calculate uplift from the mean in case of significant results. That's why id like to introduce a more robust testing framework that is able to give a solid uplift estimate. Maybe I should look at other techniques? $\endgroup$
    – ponthu
    Commented Jan 26, 2020 at 20:10
  • $\begingroup$ Also, I was thinking of calculating the delta between the observed U and the highest U (where p is .05) and using this delta to produce an uplift estimate. Would that make any sense? $\endgroup$
    – ponthu
    Commented Jan 26, 2020 at 20:17
  • $\begingroup$ Well, it may make sense to use a t-test on right- skewed data, depending on how skewed it is, and the sample size.... Comparing means may make the most sense in thinking about expected revenue. But you might consider permutation tests on the mean. $\endgroup$
    – Sal Mangiafico
    Commented Jan 26, 2020 at 22:17
  • $\begingroup$ I don't understand, and so have no opinion on the difference-in-U idea. $\endgroup$
    – Sal Mangiafico
    Commented Jan 26, 2020 at 22:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.