My software asks users for optional donations of any amount. I split test donation requests among users to find the best way to ask: 50% get request version 1, 50% get request version 2, and we see which one does better.
Almost all users give $0, but a few donate. The results might look like this:
Number of users Number of donations Dollar amounts donated GROUP A 10,000 10 40,20,20,20,15,10,10,5,5,5 GROUP B 10,000 15 50,20,10,10,10,10,10,10,5,5,5,5,5,5,5
I want to know if one group is a winner, or if it's a tie, or if we need a larger sample to be sure. (This example, kept simple for discussion, almost certainly needs a larger sample to get significant results.)
What I measure already:
- Did one group have a significantly larger number of donations? How much larger? I measure this p value and confidence interval using the ABBA Thumbtack tool, using only the number of donations and number of users, ignoring dollar amounts. Its methodology is described in the "What are the underlying statistics?" section of that link. (It's over my head, but I believe it calculates the confidence interval by taking the difference between donation rates as normal random variables on the Agresti-Couli interval.)
- Did one group donate a significantly different amount of total money? I measure this p value by performing a permutation test: repeatedly re-shuffling all 2N subjects into 2 N-subject groups, measuring the difference in total money between the groups each time, and finding the proportion of shuffles with a difference >= the observed difference. (I believe this is valid based on this this Khan Academy video doing the same thing for crackers instead of dollars.)
A few questions now about
wilcox.test() in R:
- If I fed
wilcox.test(paired=FALSE)the above table of data, would it answer any new questions not already answered by my tools above, giving me more insight with which to decide whether to keep running my test/declare a winner/declare a tie?
- If so, what exact question would it answer?