I have a problem that seems like it should be simple, but I'm having a hard time nailing down the right test. I'm analyzing sample data from a pricing experiment in which some customers are shown a higher price. The customers seeing the higher price are less likely to buy, but when they do buy they contribute more revenue. I have a dataset that tells me how much revenue was generated by each user (including those that see the price but do not buy). Here are some samples below. In real life, control and test both contain thousands of data points.

control <- (0, 0, 15, 0, 25, 0, 17, 0, 0, 12, 0)

test <- (0, 0, 0, 0, 30, 0, 19, 0, 0, 20, 0)

In this example, control had a higher conversion rate, but test generated more revenue per user. I want a test that can help me understand whether test or control are likely to generate more total revenue (or if I don't have enough data to say either way). So far I've tried running a simple t-test on the two datasets, since I was hoping that I had enough data points to not worry about the data not being normally distributed. I've also been looking at the Wilcoxon Test, but I'm having a hard time understanding it!

Sorry if this is a basic question, but any advice at all would be super helpful!

  • $\begingroup$ I'm not sure about the answer to your question, but trying to find an answer to satisfy my own curiosity, I came up with this post, which presents a fairly interesting idea: a chi-square test to test if the proportion of zeros is the same in both groups, followed by a U test for non-zero results. This doesn't apply to your problem because the price-tag for test is higher by design. But it may still be interesting to analyze separately the number of purchasing customers as a test of proportions. $\endgroup$ Nov 28, 2016 at 23:36

1 Answer 1


In your particular example, both control and test generated the same total revenue. In such a case, no statistical method will tell you whether one of the two groups will be likely to generate more in the future, or whether the (non-existent) difference is statistically significant. So I'll take the liberty of changing the control data slightly so we do have an effect we can test.

When in doubt, permute. That is, run a permutation test. In the present case, this is very simple. Throw your data together, labeled control and test. Now randomly permute these labels and calculate the difference in revenues between the two "permuted" groups. Do this many times. You now have a distribution of the difference in revenue under the null hypothesis that group membership does not make a difference. Finally, insert the actually observed difference in revenue into this distribution and check whether it is greater than, e.g., 95% of the permutation results. If so, the observed difference is statistically significant at the conventional 5% level.

# control <- c(0, 0, 15, 0, 25, 0, 17, 0, 0, 12, 0) # original data
control <- c(0, 0, 15, 0, 25, 0, 17, 0, 0, 2, 0)
test <- c(0, 0, 0, 0, 30, 0, 19, 0, 0, 20, 0)

dataset <- data.frame(
    revenue = c(control,test))
observed.difference <- diff(with(dataset,by(revenue,group,sum)))

n.perms <- 1e4
revenue.diff <- rep(NA,n.perms)
pb <- winProgressBar(max=n.perms)
for ( ii in 1:n.perms ) {
    dataset.perm <- dataset
    dataset.perm$group <- sample(dataset.perm$group,nrow(dataset.perm))
    revenue.diff[ii] <- diff(with(dataset.perm,by(revenue,group,sum)))

hist(revenue.diff,xlab="",col="grey",main="Histogram of permutation differences")


In this particular example, the actually observed difference in revenues only exceeds about 60% of the randomization results, so this would not be statistically significant at the conventional levels.

For more on permutation tests, in particular the very simple one I sketched here, I recommend Good, Permutation, Parametric, and Bootstrap Tests of Hypotheses (2005).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.