Bayesian A/B testing with uniform prior The advantage of Bayesian A/B testing is that it considers our prior belief about the distribution. However, sometimes we do not have this, so we set prior to be uniform and work with visits and conversion counts. Why would one still use Bayesian A/B testing rather than a frequentist testing?
 A: I can think of two reasons that you might want to do this:


*

*The posterior distribution $\mathbb{P}(\theta|D)$ gives us a detailed view of our knowledge about the conversation rate, $\theta$. It allows us to visualize our knowledge about the parameter's value, compare the similarity of two test variants, and use decision theory to estimate the expected value under each test.

*If we are interested in maximizing our returns while running the test, we can use methods like Thompson sampling to acquire data about which test variant is better without the test becoming too expensive. This is not always the case, but for some expensive tests it may be preferable.
A: You seem to have answered the new formulation of your question yourself, in the comments. A primary appeal of Bayesian analysis of an experiment (which is what marketers call an A/B test) is that it lets you answer probabilistic questions about population values, such as "What is the probability that treatment A is better than treatment B?". This works even if you don't have rich prior information about the effects of A and B. (You still need to choose some prior distribution, though, and your choice between various so-called uninformative priors can be surprisingly consequential.) In a frequentist approach, by contrast, the true effects of the treatments are fixed, not random, so it makes no sense to ask about the probability that one is greater than the other.
