When to terminate the Bayesian A/B test?

Question

I'm trying to do A/B testing the Bayesian way, as in Probabilistic Programming for Hackers and Bayesian A/B tests. Both articles assume that the decision maker decides which of the variants is better based solely on the probability of some criterion, e.g. $P(p_A > p_B) = 0.97 $, therefore, $A$ is better. This probability doesn't provide any information on whether there was sufficient amount of data to draw any conclusions from it. So, it is unclear to me, when to stop the test.

Suppose there are two binary RVs, $A$ and $B$, and I want to estimate how likely it is that $ p_A > p_B $, and $ \frac{p_A - p_B}{p_A} > 5\% $ based on the observations of $A$ and $B$. Additionally, suppose that $p_A$ and $p_B$ posteriors are beta-distributed.

Since I can find the $\alpha, \beta$ parameters for $p_A\,|\,\text{data} $ and $p_B\,|\,\text{data} $, I can sample the posteriors, and estimate $P(p_A > p_B\ |\ \text{data})$. Example in python:

import numpy as np

samples = {'A': np.random.beta(alpha1, beta1, 1000),
           'B': np.random.beta(alpha2, beta2, 1000)}
p = np.mean(samples['A'] > samples['B'])

I could get, for example, $P(p_A > p_B) = 0.95$. Now I would want to have something like $P(p_A > p_B\ |\ \text{data}) = 0.95 \pm 0.03$.

I have researched about credible intervals and Bayes factors, but can't understand how to calculate them for this case if they are applicable at all. How can I calculate these additional stats so that I'd have a good termination criterion?

Answer 1

I'm glad you mentioned this example, as one project I am working on is writing a whole chapter on Bayesian A/B testing.

We are interested in two quantities: $P( p_A > p_B \;|\; data)$ and some measure of "increase". I'll discuss the $P( p_A > p_B \;|\; data)$ quantity first.

There are no error bounds on $P( p_A > p_B \;|\; \text{data})$, it is a true quantity. This is similar to saying "What is the mean of the posterior?", there is only 1 mean, and we can compute it by taking the average of all the samples (I'm ignoring any Monte Carlo errors, as they can be reduced to insignificance by sampling more). I think you are mixing up unknown quantities, where we can say something like "+- 3%", and posterior-computed quantities.

What I am saying is that $P(p_A > p_B \;|\; \text{data}) = 0.95$ is certain: given your observed data and priors, this is your conclusion.

Note that we will know $p_A > p_B$ quickly: it requires only moderate amounts of observations for different enough $p_A$ and $p_B$. It is much harder, and more interesting, to measure what increase A has over B (and often this is the goal of an A/B test: how much are we increasing conversions). You mentioned that $\frac{p_A - p_B}{p_B} >$ 5% -- how certain are you of this?

Note that while $p_A > p_B$ is a boolean, and hence easy to measure,$\frac{p_A - p_B}{p_B}$ is certainly not a boolean. It is a distribution of possibilities:

As more and more data is acquired, this distribution converges to the actual relative increase, one can say the distribution stabilizes. This is where I suggest thinking about terminating the experiment. Once this distribution seems to "calm down", and we can feel confident about the increase, then terminate the experiment.

Answer 2

There seem to be two main approaches for decision making in Bayesian A/B testing. The first one is based on a paper by John Kruschke from Indiana University (K. Kruschke, Bayesian Estimation Supersedes the t Test, Journal of Experimental Psychology: General, 142, 573 (2013)). The decision rule used in this paper is based on the concept of Region Of Practical Equivalence (ROPE).

Another possibility is to use the concept of an Expected Loss. It has been proposed by Chris Stucchio (C. Stucchio, Bayesian A/B Testing at VWO). It is another approach that I would consider.

The approach suggested by Cam.Davidson.Pilon of looking at the posterior distribution of $(p_A - p_B) / p_A$ makes a lot of sense and would fit well within the ROPE method. Using the ROPE method has the added advantage of giving also a rule for when the experiment is inconclusive (not just when the "A" or "B" variants can be declared winners).

You can find more in this blog post: Bayesian A/B Testing: a step-by-step guide. It also includes some Python code snippets that are mostly based on a Python project hosted on Github.

Answer 3

I've been experimenting with ways to stop a Bayesian A/B test and you're right - there aren't that many obvious ways from googling around. The method I like most is a precision based method, based on this: http://doingbayesiandataanalysis.blogspot.com/2013/11/optional-stopping-in-data-collection-p.html. However, I haven't found much mathematical literature around this, so right now it's just a good heuristic.

I've found that while my tests need to run much longer in order to hit some desired precision, it's more intuitive and you're giving time for the distribution of $P(A > B | data)$ to "calm down" in an objective way, i.e. rather than eye-balling it.