I'm currently running an AB test on a bunch of websites for a change we've made on all the websites and am measuring revenue/site/group so my test groups look like so:
Website a | Test group a | unique visitors | revenue Website a | Test group b | unique visitors | revenue Website b | Test group a | unique visitors | revenue Website b | Test group b | unique visitors | revenue
Initially I want to show there's a difference between the two test groups overall and so have used a null hypothesis of: "The test group makes no difference to the revenue/cookie across all sites".
I can then happily sum all the unique visitors (across sites) and revenue for each test group. However, I don't really know what to do from here.
I know that conversions can be treated binomially but revenue can't be. We also can't treat average order value as an exponential or normal distribution (ideas I looked at). Initially, I thought about using a Mann-Whitney U test but most things I've read seems to suggest these work best for < 20 entries in your array.
When I create a vector containing the amount every customer spent (a very sparsely populated vector - most people don't buy anything), each test group has a different length vector of size ~100,000.
My next attempt involved bootstrapping (I think) - I basically created a normal distribution for each test group by sampling from the aforementioned sparsely populated revenue vector. Creating around 10,000 vectors like this and running a normality test on the sum of each of the vectors gives me a normal distribution for each test group. From this I believe I can perform a t-test on the two normally distributed vectors to find the significance of the change made and hence support/disprove the null hypothesis?
My question is, are either of the above attempts the right thing to do? I can't imagine that this is that uncommon a thing to want to do that there isn't a generally accepted way of doing this. Additionally, I'd also like to be able to report something along the lines of:
Test group x is y% better than test group z with 5th and 95th percentile on that percentage uplift of q and j.
What would be the best way to calculate these bits of information?