Bayesian A/B/C testing

Question

I built a Bayesian A/B testing tool - i.e. one which models A and B having posteriors $Beta(\alpha_i, \beta_i)$ where $\alpha_i, \beta_i$ are updated every iteration.

After T iterations, I compare the posterior distributions of A and B and determine which one is larger, lets say, using the mean.

How can I incorporate a third (or fourth...) group?

Originally I wanted to model it as a Dirichlet, giving each group some $\alpha$ that is updated each iteration. And then after T iterations select the group with the largest $\alpha$. But I think that it's more correct to keep each group A, B, C as a separate Beta (since they are theoretically independent) and just compare the posteriors of their respective distributions. It seems naive to me, but also intuitive. Can someone who has done this please opine?

Answer 1

Addressing your question

As you have stated correctly, the groups are independent, hence Dirichlet does not make any sense. When you have more than one group, you can boil the comparison down to 2 by one of these strategies:

• You have a reference / baseline group, which shall be improved. Compare all groups to that one.
• Compare the current best to the current second best to decide whether you have a global winner. In the same way, compare the worst to the second-worst to decide whether a group shall dropped from the competition.

General remark

Note: Maybe I have misunderstood what you meant with "comparison by mean", I just add this for sake of completeness for other readers.

Comparing the posteriors is good, but the current mean is rather volatile and hence may be bigger or smaller just by chance. IMHO it is recommended to compare the credible intervals of the means using a two-dimensional monte carlo integration to calculate the subjective probability of the true unknown mean of group B is greater than the true unknown mean of group A, i.e. group B > group A.

Example code for

• A: 401 trials, 125 successful trials
• B: 441 trials, 141 successful trials

which is

trials <- 10000
resDat<-data.frame("a"=rbeta(trials,125+1,401-125+1),
                   "b"=rbeta(trials,144+1,441-144+1))
length(which(resDat$b>resDat$a))/trials

But even using this approach one does only control the type II error, i.e. falsely missing a difference. To control the type I error, i.e. falsely stating finding a difference, I recommend a classical frequentist style test (e.g. G-Test) as followup.

Answer 2

The Beta distribution could be seen as a special case of the Dirichlet distribution and the question is technically not about whether you are using a Dirichlet or not (because a Beta distribution is already a Dirichlet distribution), but the question is about which dimension to use.

These Dirichlet distributions are conjugate priors for the categorical distribution. In the case of the Beta distribution, you have just two categories, and in the case of the Dirichlet distribution, you have more categories.

Which categories and categorical distribution are you modeling with A/B testing?

With A/B testing you are not modeling a probability distribution for probabilities among the two groups/categories A and B, which now become three groups with the addition of C. (this is where your intuition about the need to use a Dirichlet distribution may come from)

Instead, with A/B testing you are more likely* modeling something like success rate and the beta distribution posterior relates to the two classes 'success' and 'failure'.

---

^{* I write, more likely. Since you are using a beta distribution as prior/posterior, I assume that you are probably modelling something that is binomial distributed like click-through rate, or number of visitors that continue to make a successful sale. But A/B testing is more general and can also relate to other distributions. E.g. it could be about the total value of sales instead of the number of sales.}