# Calculating significance and uplift on Revenue A/B Tests

I'm currently running an AB test on a bunch of websites for a change we've made on all the websites and am measuring revenue/site/group so my test groups look like so:

Website a | Test group a | unique visitors | revenue
Website a | Test group b | unique visitors | revenue
Website b | Test group a | unique visitors | revenue
Website b | Test group b | unique visitors | revenue


etc.

Initially I want to show there's a difference between the two test groups overall and so have used a null hypothesis of: "The test group makes no difference to the revenue/cookie across all sites".

I can then happily sum all the unique visitors (across sites) and revenue for each test group. However, I don't really know what to do from here.

I know that conversions can be treated binomially but revenue can't be. We also can't treat average order value as an exponential or normal distribution (ideas I looked at). Initially, I thought about using a Mann-Whitney U test but most things I've read seems to suggest these work best for < 20 entries in your array.

When I create a vector containing the amount every customer spent (a very sparsely populated vector - most people don't buy anything), each test group has a different length vector of size ~100,000.

My next attempt involved bootstrapping (I think) - I basically created a normal distribution for each test group by sampling from the aforementioned sparsely populated revenue vector. Creating around 10,000 vectors like this and running a normality test on the sum of each of the vectors gives me a normal distribution for each test group. From this I believe I can perform a t-test on the two normally distributed vectors to find the significance of the change made and hence support/disprove the null hypothesis?

My question is, are either of the above attempts the right thing to do? I can't imagine that this is that uncommon a thing to want to do that there isn't a generally accepted way of doing this. Additionally, I'd also like to be able to report something along the lines of:

Test group x is y% better than test group z with 5th and 95th percentile on that percentage uplift of q and j.

What would be the best way to calculate these bits of information?

• I'm doing this exact same thing and using Bayesian statistics to form credibility intervals for my lift. I run a Bayesian bootstrap on my revenue per order metric and then sample from a dirichlet and build a distribution of sampled means for control and each variation as well as lift of each variation over control. So it sounds like you're heading down the right path. – Justin Bozonier Nov 12 '14 at 3:36
• Oh also, unless your revenue per order is very large I would be very surprised to see a normal distribution. I'd expect it to be skewed right and look log-normal or so. Verify that assumption. Having said that, if you're bootstrapping you can expect a distribution of sampled means of your metric to probably be normal. The tests you want to run may make sense there. – Justin Bozonier Nov 12 '14 at 3:38