For context: How to define prior for beta-binomial A/B test

For P(A > B), you can draw samples from A's posterior and B's posterior and then count the number of times the sample from A is greater than the sample from B. If choice C entered the picture, I don't intuitively see why:

mean(A_samp > B_samp & A_samp > C_samp)

wouldn't work, but I suspect that's a bit too simplistic. I'm not sure how to extend this to the n >= 3 case.


Actually, this blog entry by Evan Miller (mentioned in the link you refer to) describes case when you want to test the hypothesis that $A > \max(B,C)$ for binary and count data. It basically gives closed-form solution for what you are suggesting, so in fact there is not need to conduct simulation.

Notice however that since people mean often different things by A/B testing and if you mean that your data is counts of choices in multiple-choices scenario, then Dirichlet-multinomial model would be more appropriate.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.