Is the Wilcoxon rank-sum test the right test to see if total donations differ?

Question

Background:

My software asks users for optional donations of any amount. I split test donation requests among users to find the best way to ask: 50% get request version 1, 50% get request version 2, and we see which one does better.

Almost all users give $0, but a few donate. The results might look like this:

         Number of users  Number of donations   Dollar amounts donated
GROUP A  10,000           10                    40,20,20,20,15,10,10,5,5,5
GROUP B  10,000           15                    50,20,10,10,10,10,10,10,5,5,5,5,5,5,5

I want to know if one group is a winner, or if it's a tie, or if we need a larger sample to be sure. (This example, kept simple for discussion, almost certainly needs a larger sample to get significant results.)

What I measure already:

1. Did one group have a significantly larger number of donations? How much larger? I measure this p value and confidence interval using the ABBA Thumbtack tool, using only the number of donations and number of users, ignoring dollar amounts. Its methodology is described in the "What are the underlying statistics?" section of that link. (It's over my head, but I believe it calculates the confidence interval by taking the difference between donation rates as normal random variables on the Agresti-Couli interval.)
2. Did one group donate a significantly different amount of total money? I measure this p value by performing a permutation test: repeatedly re-shuffling all 2N subjects into 2 N-subject groups, measuring the difference in total money between the groups each time, and finding the proportion of shuffles with a difference >= the observed difference. (I believe this is valid based on this this Khan Academy video doing the same thing for crackers instead of dollars.)

R's wilcox.test:

A few questions now about wilcox.test() in R:

1. If I fed wilcox.test(paired=FALSE) the above table of data, would it answer any new questions not already answered by my tools above, giving me more insight with which to decide whether to keep running my test/declare a winner/declare a tie?
2. If so, what exact question would it answer?

Answer 1

If you use wilcox.test() with the argument paired (note that this is lower case, and that R is case sensitive) set to FALSE, you are running a Mann-Whitney $U$-test. This is a test of stochastic dominance. If the distributions were equal, and you picked an observation from each version at random, the observation from version 2 would have a 50%-50% chance of being higher than the observation from version 1. On the other hand, the value drawn from version 2 might have a greater than 50% chance of being greater than (less than) the value from version 1. This is stochastic dominance. Nothing is being said about how much greater or less, only that it is greater or lesser.

That doesn't strike me as a good fit for your objectives. You want the most total money, which can be understood as the largest mean donation times the number of users. It is possible, due to skew, that one version can have the largest mean / total, but that the other version is stochastically greater. (If that were the case, you would want the former version.) Because this is what you ultimately want, a test that is specific to that aspect of the distributions is most appropriate for you.

I recognize that your data are not remotely normal, and thus, the $t$-test (which might be what most people would think of first for comparing two groups), would be inappropriate. Given two continuous, but non-normal groups, most people might likewise automatically go with the Mann-Whitney. In your case, I would go with a permutation test, for the above reason. (I gather this is what you did, if I understood correctly.) A permutation test is valid here, because you randomly assigned users to the two groups; therefore, they are exchangeable.

To perform a permutation test, just shuffle the grouping indicator and calculate means and a difference between the means. Doing this many times will allow you to create a sampling distribution of the difference between the means. You can compare your observed difference to the sampling distribution. For a two-tailed test, take the smaller proportion beyond your difference and multiply it by two. The product is directly interpretable as a $p$-value. Here is a worked example with your data:

A            = c(rep(0, 9990), 40,20,20,20,15,10,10,5,5,5)
B            = c(rep(0, 9985), 50,20,10,10,10,10,10,10,5,5,5,5,5,5,5)
realized.dif = mean(B)-mean(A);  realized.dif  # [1] 0.0015

set.seed(6497)
donations = stack(list(A=A, B=B))
values    = donations$values
ind       = donations$ind
difs      = vector(length=1000)
for(i in 1:1000){
  ind     = sample(ind)
  difs[i] = mean(values[ind=="B"])-mean(values[ind=="A"])
}
difs = sort(difs)
mean(difs>=realized.dif)    # [1] 0.459  # a 1-tailed test, if Ha: B>A a-priori
mean(difs>=realized.dif)*2  # [1] 0.918  # a 2-tailed test

---

Regarding the first study question, i.e., 'which version yielded a larger number of donations', while I grant that everybody loves ABBA, you can do this in R as well. I would use a $z$-test of the difference of two proportions. In R, that's prop.test(). Here is an example using your data:

prop.test(rbind(c(10, 9990),
                c(15, 9985) ))
#  2-sample test for equality of proportions with continuity correction
# 
# data:  rbind(c(10, 9990), c(15, 9985))
# X-squared = 0.6408, df = 1, p-value = 0.4234
# alternative hypothesis: two.sided
# 95 percent confidence interval:
#   -0.0015793448  0.0005793448
# sample estimates:
# prop 1 prop 2 
# 0.0010 0.0015 

Answer 2

@gung's answer is correct. But I would add that since your data might be skewed, with a huge right tail, the mean may not be robust and as such it might not be the "right" index for representing the centrality of your distribution. Hence, I would try also with more robust solutions such as medians or truncated means.