7
$\begingroup$

I'm currently running an AB test on a bunch of websites for a change we've made on all the websites and am measuring revenue/site/group so my test groups look like so:

Website a | Test group a | unique visitors | revenue
Website a | Test group b | unique visitors | revenue
Website b | Test group a | unique visitors | revenue
Website b | Test group b | unique visitors | revenue

etc.

Initially I want to show there's a difference between the two test groups overall and so have used a null hypothesis of: "The test group makes no difference to the revenue/cookie across all sites".

I can then happily sum all the unique visitors (across sites) and revenue for each test group. However, I don't really know what to do from here.

I know that conversions can be treated binomially but revenue can't be. We also can't treat average order value as an exponential or normal distribution (ideas I looked at). Initially, I thought about using a Mann-Whitney U test but most things I've read seems to suggest these work best for < 20 entries in your array.

When I create a vector containing the amount every customer spent (a very sparsely populated vector - most people don't buy anything), each test group has a different length vector of size ~100,000.

My next attempt involved bootstrapping (I think) - I basically created a normal distribution for each test group by sampling from the aforementioned sparsely populated revenue vector. Creating around 10,000 vectors like this and running a normality test on the sum of each of the vectors gives me a normal distribution for each test group. From this I believe I can perform a t-test on the two normally distributed vectors to find the significance of the change made and hence support/disprove the null hypothesis?

My question is, are either of the above attempts the right thing to do? I can't imagine that this is that uncommon a thing to want to do that there isn't a generally accepted way of doing this. Additionally, I'd also like to be able to report something along the lines of:

Test group x is y% better than test group z with 5th and 95th percentile on that percentage uplift of q and j.

What would be the best way to calculate these bits of information?

Improve this question
$\endgroup$
2
  • $\begingroup$ I'm doing this exact same thing and using Bayesian statistics to form credibility intervals for my lift. I run a Bayesian bootstrap on my revenue per order metric and then sample from a dirichlet and build a distribution of sampled means for control and each variation as well as lift of each variation over control. So it sounds like you're heading down the right path. $\endgroup$
    – Justin Bozonier
    Commented Nov 12, 2014 at 3:36
  • $\begingroup$ Oh also, unless your revenue per order is very large I would be very surprised to see a normal distribution. I'd expect it to be skewed right and look log-normal or so. Verify that assumption. Having said that, if you're bootstrapping you can expect a distribution of sampled means of your metric to probably be normal. The tests you want to run may make sense there. $\endgroup$
    – Justin Bozonier
    Commented Nov 12, 2014 at 3:38

2 Answers 2

Reset to default
1
$\begingroup$

As I understand you have a 2X2 experimental design ( Factor1 - website(levels: a,b), factor2 - visitors group (levels: a,b)) and the dependent variable 'Revenue'.

I would consider ANOVA being careful to all assumptions behind. The way you measure/code dependent variable 'Revenue' has different implications.
This movie from Andy Field might be useful. Regards, Marius

Improve this answer
$\endgroup$
1
$\begingroup$

Probably the revenue per cookie has a normal distribution (you can check this with bootstrapping). The revenue per si do not have a normal distribution, just the revenue per cookie. That said, you can applied an hypothesis test as usually an check is the difference in revenues per user in two groups are significant.

Other approach is considers the revenue distribution as a combination of two others distribution. The first one is the distribution for the conversion (buy or not) and the other distribution if for the average revenue per order.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.