Bayesian AB testing

Question

I am running an AB Test on a page that receives only 5k visits per month. It would take too long to reach traffic levels necessary to measure a +-1% difference between the test and control. I have heard that I can use Bayesian stats to give me a good chance of determining whether the test outperformed. How can I do use Bayesian stats to analyze my current data?

        Visitors    Conversions
Control 1345         1165
Test A  961          298
Test B  1274         438

Answer 1

You can perform a Monte-Carlo-Integration of the credible intervals of each group represented by beta distributions to calculate the probability that the true unknown parameter of one group is better than the true unknown parameter of another group. I've done something similar in this question How does a frequentist calculate the chance that group A beats group B regarding binary response where trials=Visitors and successful trials = conversions

BUT: Beware that Bayes will give you only subjective probabilities depending on the data collected so far, not the objective "truth". This is rooted in the difference in philosophy between frequentists (which use statistical tests, p-values etc) and Bayesians. Hence you cannot expect to detect a significant difference using Bayes when the statistical procedures fail to do so.

To understand why this matters it might help to learn the difference between the confidence interval and the credible interval first, since the above mentioned MC-Integration "only" compares two indepent credible intervals with each other.

For further details on this topic see e.g. this questions:

• Bayesian and Frequentist reasoning in Plain English
• What, precisely, is a confidence interval?
• What's the difference between a confidence interval and a credible interval?

Answer 2

I'm working my way through the same questions. There are now a couple of helpful articles that weren't available when you posed this question.

• Bayesian A/B testing with theory & code
• Proportionate A/B Testing