I built a Bayesian A/B testing tool - i.e. one which models A and B having posteriors $Beta(\alpha_i, \beta_i)$ where $\alpha_i, \beta_i$ are updated every iteration.

After T iterations, I compare the posterior distributions of A and B and determine which one is larger, lets say, using the mean.

How can I incorporate a third (or fourth...) group?

Originally I wanted to model it as a Dirichlet, giving each group some $\alpha$ that is updated each iteration. And then after T iterations select the group with the largest $\alpha$. But I think that it's more correct to keep each group A, B, C as a separate Beta (since they are theoretically independent) and just compare the posteriors of their respective distributions. It seems naive to me, but also intuitive. Can someone who has done this please opine?


Addressing your question

As you have stated correctly, the groups are independent, hence Dirichlet does not make any sense. When you have more than one group, you can boil the comparison down to 2 by one of these strategies:

  • You have a reference / baseline group, which shall be improved. Compare all groups to that one.
  • Compare the current best to the current second best to decide whether you have a global winner. In the same way, compare the worst to the second-worst to decide whether a group shall dropped from the competition.

General remark

Note: Maybe I have misunderstood what you meant with "comparison by mean", I just add this for sake of completeness for other readers.

Comparing the posteriors is good, but the current mean is rather volatile and hence may be bigger or smaller just by chance. IMHO it is recommended to compare the credible intervals of the means using a two-dimensional monte carlo integration to calculate the subjective probability of the true unknown mean of group B is greater than the true unknown mean of group A, i.e. group B > group A.

Example code for

  • A: 401 trials, 125 successful trials
  • B: 441 trials, 141 successful trials

which is

trials <- 10000

But even using this approach one does only control the type II error, i.e. falsely missing a difference. To control the type I error, i.e. falsely stating finding a difference, I recommend a classical frequentist style test (e.g. G-Test) as followup.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.