I am running an A/B test where I submit website visitors to one of two destinations. If the visitor converts on either of the destinations the payout is between
$100. Conversions happen approximately 20% of the time. The split is random approximately
80% to the control group and
20% to the test group.
I am trying to establish if one destination performs better than the other by looking at revenue per submit.
Because the payouts are nonparametric I initially tried to use the Mann-Whitney-Wilcoxon test but I found that due to the number of observations I have the results always came back significant.
For as common as I imagined this task would be I've been unable to find any resources that address it.
I was wondering if there is a best practice on how to deal with tests like this? Is Mann-Whitney-Wilcoxon the best approach? If so, what is the best way to deal with too much data resulting in always-significant results?