1
$\begingroup$

I am running an A/B test where I submit website visitors to one of two destinations. If the visitor converts on either of the destinations the payout is between $1 and $100. Conversions happen approximately 20% of the time. The split is random approximately 80% to the control group and 20% to the test group.

I am trying to establish if one destination performs better than the other by looking at revenue per submit.

Because the payouts are nonparametric I initially tried to use the Mann-Whitney-Wilcoxon test but I found that due to the number of observations I have the results always came back significant.

For as common as I imagined this task would be I've been unable to find any resources that address it.

I was wondering if there is a best practice on how to deal with tests like this? Is Mann-Whitney-Wilcoxon the best approach? If so, what is the best way to deal with too much data resulting in always-significant results?

Improve this question
$\endgroup$
1
  • $\begingroup$ I'm not sure why you think it's a problem that the test rejects the null hypothesis. Significance tests can only disconfirm the null hypothesis, not show that the effect is big enough to be important, or something. In real life, the null hypothesis is rarely if ever true, so all the effects in large samples are significant. Are you sure that significance testing is of any use to you? $\endgroup$
    – Kodiologist
    Jul 12, 2017 at 18:52

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.