My software asks users for optional donations of any amount. I split test donation requests among users to find the best way to ask: 50% get request version 1, 50% get request version 2, and we see which one does better.

Almost all users give $0, but a few donate. The results might look like this:

         Number of users  Number of donations   Dollar amounts donated
GROUP A  10,000           10                    40,20,20,20,15,10,10,5,5,5
GROUP B  10,000           15                    50,20,10,10,10,10,10,10,5,5,5,5,5,5,5

I want to know if one group is a winner, or if it's a tie, or if we need a larger sample to be sure. (This example, kept simple for discussion, almost certainly needs a larger sample to get significant results.)

What I measure already:

  1. Did one group have a significantly larger number of donations? How much larger? I measure this p value and confidence interval using the ABBA Thumbtack tool, using only the number of donations and number of users, ignoring dollar amounts. Its methodology is described in the "What are the underlying statistics?" section of that link. (It's over my head, but I believe it calculates the confidence interval by taking the difference between donation rates as normal random variables on the Agresti-Couli interval.)
  2. Did one group donate a significantly different amount of total money? I measure this p value by performing a permutation test: repeatedly re-shuffling all 2N subjects into 2 N-subject groups, measuring the difference in total money between the groups each time, and finding the proportion of shuffles with a difference >= the observed difference. (I believe this is valid based on this this Khan Academy video doing the same thing for crackers instead of dollars.)

R's wilcox.test:

A few questions now about wilcox.test() in R:

  1. If I fed wilcox.test(paired=FALSE) the above table of data, would it answer any new questions not already answered by my tools above, giving me more insight with which to decide whether to keep running my test/declare a winner/declare a tie?
  2. If so, what exact question would it answer?
  • $\begingroup$ Could you describe what the two things you're currently doing are in the question, rather than relying on links? The second one in particular links to a video, which I, and expect others, are not going to take the time to watch to figure out what you're asking. $\endgroup$ – Aaron Feb 10 '15 at 21:09
  • 1
    $\begingroup$ This question is clearly about understanding how the different tests relate to study goals. It barely has any connection to coding. It is on-topic here, & would be off-topic on Stack Overflow. $\endgroup$ – gung Feb 10 '15 at 22:01
  • $\begingroup$ @Aaron: done. Thanks for the feedback. I was afraid the wall of text would dissuade people from reading the question in the first place. Hard to split test my qusetions to optimize for answers on CrossValidated ;) $\endgroup$ – Michael Gundlach Feb 10 '15 at 23:31

If you use wilcox.test() with the argument paired (note that this is lower case, and that R is case sensitive) set to FALSE, you are running a Mann-Whitney $U$-test. This is a test of stochastic dominance. If the distributions were equal, and you picked an observation from each version at random, the observation from version 2 would have a 50%-50% chance of being higher than the observation from version 1. On the other hand, the value drawn from version 2 might have a greater than 50% chance of being greater than (less than) the value from version 1. This is stochastic dominance. Nothing is being said about how much greater or less, only that it is greater or lesser.

That doesn't strike me as a good fit for your objectives. You want the most total money, which can be understood as the largest mean donation times the number of users. It is possible, due to skew, that one version can have the largest mean / total, but that the other version is stochastically greater. (If that were the case, you would want the former version.) Because this is what you ultimately want, a test that is specific to that aspect of the distributions is most appropriate for you.

I recognize that your data are not remotely normal, and thus, the $t$-test (which might be what most people would think of first for comparing two groups), would be inappropriate. Given two continuous, but non-normal groups, most people might likewise automatically go with the Mann-Whitney. In your case, I would go with a permutation test, for the above reason. (I gather this is what you did, if I understood correctly.) A permutation test is valid here, because you randomly assigned users to the two groups; therefore, they are exchangeable.

To perform a permutation test, just shuffle the grouping indicator and calculate means and a difference between the means. Doing this many times will allow you to create a sampling distribution of the difference between the means. You can compare your observed difference to the sampling distribution. For a two-tailed test, take the smaller proportion beyond your difference and multiply it by two. The product is directly interpretable as a $p$-value. Here is a worked example with your data:

A            = c(rep(0, 9990), 40,20,20,20,15,10,10,5,5,5)
B            = c(rep(0, 9985), 50,20,10,10,10,10,10,10,5,5,5,5,5,5,5)
realized.dif = mean(B)-mean(A);  realized.dif  # [1] 0.0015

donations = stack(list(A=A, B=B))
values    = donations$values
ind       = donations$ind
difs      = vector(length=1000)
for(i in 1:1000){
  ind     = sample(ind)
  difs[i] = mean(values[ind=="B"])-mean(values[ind=="A"])
difs = sort(difs)
mean(difs>=realized.dif)    # [1] 0.459  # a 1-tailed test, if Ha: B>A a-priori
mean(difs>=realized.dif)*2  # [1] 0.918  # a 2-tailed test

Regarding the first study question, i.e., 'which version yielded a larger number of donations', while I grant that everybody loves ABBA, you can do this in R as well. I would use a $z$-test of the difference of two proportions. In R, that's prop.test(). Here is an example using your data:

prop.test(rbind(c(10, 9990),
                c(15, 9985) ))
#  2-sample test for equality of proportions with continuity correction
# data:  rbind(c(10, 9990), c(15, 9985))
# X-squared = 0.6408, df = 1, p-value = 0.4234
# alternative hypothesis: two.sided
# 95 percent confidence interval:
#   -0.0015793448  0.0005793448
# sample estimates:
# prop 1 prop 2 
# 0.0010 0.0015 
  • $\begingroup$ Nice answer, and +2 if I could for the ABBA link. :) $\endgroup$ – Aaron Feb 11 '15 at 14:32
  • $\begingroup$ +1 Good discussion, especially about comparing means. Note that because there are no huge donations, the example in the question has a nearly normal sampling distribution. The count of donations of a given value in group A is Binomial and, to high accuracy, all such counts are approximately independent. The t-test should work well. Here is a plot of the null distribution: b <- function(n) dbinom(0:n, n, 1/2); p <- apply(expand.grid(b(1), b(1), b(4), b(6), b(10)), 1, prod); n <- as.matrix(expand.grid(0:1, 0:1, 0:4, 0:6, 0:10)) %*% c(50,40,20,10,5); plot(dist <- aggregate(p, list(n), sum)). $\endgroup$ – whuber Feb 11 '15 at 20:18
  • 1
    $\begingroup$ I'm not being critical, but here's where I'm going: by examining the highest few donations you can quickly decide whether to use a t-test or not. If you cannot, the difference in means will be strongly dependent on those few outliers and so is unlikely to be significant. These observations simplify the analysis and they also suggest one could rely on normal-theory-based sample size calculations to estimate how large a sample would be needed. Even better, though, would be to use a sequential sampling design. $\endgroup$ – whuber Feb 11 '15 at 20:32
  • 1
    $\begingroup$ @Torvon for permutation testing? There are lots, but I doubt you need one--this is a pretty well-established & common technique. $\endgroup$ – gung Feb 24 '15 at 15:02
  • 1
    $\begingroup$ @Torvon, instead of simply calculating the mean difference & storing it, run a M-W U & store it. $\endgroup$ – gung Mar 6 '15 at 15:04

@gung's answer is correct. But I would add that since your data might be skewed, with a huge right tail, the mean may not be robust and as such it might not be the "right" index for representing the centrality of your distribution. Hence, I would try also with more robust solutions such as medians or truncated means.

  • 4
    $\begingroup$ One of @gung's most important points is that the median and truncated means are likely irrelevant. (See the paragraph on "good fit for your objectives.") We care most about the mean donation. So, although your robust solutions may be procedurally simple to use, getting the right answer to the wrong question might be of little help--or even worse. $\endgroup$ – whuber Feb 11 '15 at 20:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.