How to define prior for beta-binomial A/B test

Question

I would like to run an A/B test using a Bayesian beta-binomial model whereby I would state probabilities such as $P(p_B>p_A)$ in place of using a traditional T-test. I've read that the prior should be constructed as $f(a,b) \sim (a+b)^{-\frac{5}{2}}$. How exactly is this done? I'm looking for an actual example of this step of the modeling process.

Here is my A/B test data by day formatted as: date, A enrollments, A conversions, B enrollments, B conversions.

10/1/2015,150,16,148,17
10/2/2015,163,17,165,19
10/3/2015,155,14,157,18
10/4/2015,161,15,153,19
10/5/2015,148,14,150,16

Answer 1

I'm not sure what that prior represents, but from what you're looking to do, you should set up some Beta prior with your Bernoulli likelihood.

The posterior distribution of a Beta-Bernoulli is: $$\sim Beta (a_0 + \sum{x_i} , b_0 + n - \sum{x_i})$$

where $\frac{a_0}{a_0 + b_0}$ is your prior conversion rate. The size of $a_{0}$ and $b_0$ represent the strength of belief in your prior. And $\sum{x_i}$ is the number of conversions you observe for a given day.

Now how you want to model it is up to you. You could model a conversion rate per day, or a cumulative conversion rate. You'll want to update the model for new data that comes in each day. You also must assume that each data point is independent of the others, which is a strong (perhaps invalid) assumption, especially if you have repeat visitors.

In the end you'll have a posterior distribution for $A$ and $B$. Then you could sample from those posteriors and count how many times $B > A$, which is your $P(B>A)$. You could also provide other posterior summaries, perhaps calculate a credible interval for $B - A$, or do any number of Bayesian posterior summaries. Also don't forget to calculate the power of the test to ensure you have a sufficient sample size.

Answer 2

In beta-binomial model beta distribution is a prior for binomial distribution. It is co-called conjugate prior, i.e. there exists closed form for beta-binomial distribution.

If you recall, Bayes theorem states that to obtain posterior we need likelihood function and a prior

$$ \underbrace{P(\theta|D)}_\text{posterior} \propto \underbrace{P(D|\theta)}_\text{likelihood} \times \underbrace{P(\theta)}_\text{prior} $$

then in beta-binomial model you use binomial distribution as a likelihood function and beta distribution as prior for $p$ parameter in binomial distribution, i.e.

$$ x_i \sim \mathrm{Binomial}(n, p) $$ $$ p \sim \mathrm{Beta}(\alpha, \beta) $$

where you need to assume some $\alpha$ and $\beta$ parameters for beta. If you want to assume that each value of $p$ has the same probability, i.e. you have no out-of-data (prior) information that you would like to include in your model, that you can set $\alpha = \beta = 1$, that is uniform for all the values of $p$ in $[0,1]$ interval. You can look at this thread for understanding beta distribution better, so it is easier for you to choose other values of $\alpha$ and $\beta$ if needed.

You can find code example of such A/B test, you cal look at this thread. If you are looking for examples, I would recommend this blog entry by Rasmus Bååth on binomial test implemented in Bayesian First Aid package for R. This topic is also covered in probably any handbook on Bayesian data analysis, e.g. Gelman eta al. (2004), or Kruschke (2010). Google search also reveals multiple worked examples of such model (e.g. this blog for introductory example).

---

Gelman, A., Carlin, J.B., Stern, H.S., and Rubin, D.B. (2014). Bayesian data analysis. London: Chapman & Hall/CRC.

Kruschke, J. (2010). Doing Bayesian data analysis: A tutorial introduction with R. Academic Press.