I am running an AB Test on a page that receives only 5k visits per month. It would take too long to reach traffic levels necessary to measure a +-1% difference between the test and control. I have heard that I can use Bayesian stats to give me a good chance of determining whether the test outperformed. How can I do use Bayesian stats to analyze my current data?
Visitors Conversions
Control 1345 1165
Test A 961 298
Test B 1274 438
You can perform a Monte-Carlo-Integration of the credible intervals of each group represented by beta distributions to calculate the probability that the true unknown parameter of one group is better than the true unknown parameter of another group. I've done something similar in this question How does a frequentist calculate the chance that group A beats group B regarding binary response where trials=Visitors and successful trials = conversions
BUT: Beware that Bayes will give you only subjective probabilities depending on the data collected so far, not the objective "truth". This is rooted in the difference in philosophy between frequentists (which use statistical tests, p-values etc) and Bayesians. Hence you cannot expect to detect a significant difference using Bayes when the statistical procedures fail to do so.
To understand why this matters it might help to learn the difference between the confidence interval and the credible interval first, since the above mentioned MC-Integration "only" compares two indepent credible intervals with each other.
For further details on this topic see e.g. this questions:
I'm working my way through the same questions. There are now a couple of helpful articles that weren't available when you posed this question.
"Bayesian A/B testing with theory and code" by Antti Rasinen - the logical conclusion of an unfinished series of articles series "Exact Bayesian Inference for A/B testing" by Evan Haas (partially rescued here part1 and part2).
The conjugate prior for the binomial distribution is the beta distribution. Therefore the distribution of the conversion rate for one variant is the beta distribution. You can solve $Pr(A > B)$ numerically or exactly. The author refers to an essay written by Bayes himself, "An Essay towards solving a Problem in the Doctrine of Chances".
"Proportionate A/B Testing" by Ian Clarke - Author explains that the beta distribution is the key to understanding how to apply a Bayesian solution to A/B testing. He also discusses the use of Thompson Sampling for determining prior values for $\alpha$ and $\beta$.
"Chapter 2: A little more on PyMC" from the book "Bayesian Methods for Hackers" by Cam Davidson Pilon - This is an iPython book explaining Bayesian methods in a number of applications. About half way through Chapter 2 (the section title is Example: Bayesian A/B testing), the author gives a detailed explanation of how to calculate probability that A is better than B (or vice versa) using the pymc library. Full python code is given, including plotting the results.
There are also now a number of Bayesian significance calculators online as well:
There are several approaches for doing Bayesian A/B testing.
First of all, you should decide whether you want to use an analytic approach (using conjugate distributions as Lenwood mentions) or an MCMC approach. For simple A/B experiments, particularly on conversion rate which is your case, there is really no need to use an MCMC approach: just use a Beta distribution as a prior and your posterior distribution will also be a Beta distribution.
Then, you need to decide which decision rule to apply. Here, there seems to be two main approaches for decision making. The first one is based on a paper by John Kruschke from Indiana University (K. Kruschke, Bayesian Estimation Supersedes the t Test, Journal of Experimental Psychology: General, 142, 573 (2013).). The decision rule used in this paper is based on the concept of Region Of Practical Equivalence (ROPE).
Another possibility is to use the concept of an Expected Loss. It has been proposed by Chris Stucchio (C. Stucchio, Bayesian A/B Testing at VWO).
In principle, you could use a different decision rule.
You can find this and much more on this blog post: Bayesian A/B Testing: a step-by-step guide. It also includes some Python code snippets and uses a Python project that is hosted on Github.
