I was reading the book Probabilistic Programming for Hackers and came across Bayesian A/B Testing. So basically, I get a Posterior Distribution for $P(A), P(B)$ and the $\Delta$ between $P(A)$ and $P(B)$.

How is this different from a Frequentist statistics? I mean I can just simply estimate the $P(A)$ and $P(B)$ using frequentist methods an I also get confidence intervals around it. I can't see the benefit of Bayesian method here.


1 Answer 1


The benefit of bayesianism is in the ability to add prior information to your model. So, for instance, say you introduce a new variant to your platform and will measure the performance of the new variant against a control group (which sees the site as usual).

Well, you know how the control group should act because you've observed how people act on the site as it is. You can put this info into your model and this get away with a smaller sample size for the control group because of this.

As for the confidence intervals, those are based on asymptotic assumptions about the sampling distribution of the statsitic of interest. It is hard to say when or if those asymptotic results kick in, especially when samples are not very large. In Bayesianism, we trade the asymptotic assumptions for assumptions about ergodicity of markov chains as they sample the posterior distribution. Is one set of assumptions "better" or "more justifiable" than the other? No.

There are also other benefits. The assumptions of Bayesianism are easier to criticize because modelers have to explicitly write down all of their assumptions. For instance, if you assume in a bayesian linear regression that there is a single parameter that specifies the variance of the outcome, then it is very clear you are assuming heterogeneity of variance.

Bayesianism also has obstacles. Priors are easy to "hack" in the same way that frequentism is easy to p-hack. Not to mention the computational effort required to run modestly complex Bayesian models can be quite high (though great strides are being made in these avenues. Stan can run Bayesian models very very fast as they have made some very good optimizations to the way they run the HMC).

Frankly, for AB tests, frequentism is just fine unless you are making more complex assumptions about the data generating process. The frequency properties of the test of proportions are well studied, and if your goal is to analyze a simple AB test, then don't fix what ain't broken.

  • $\begingroup$ Thank you. Well explained! I really appreciate your answer. I'm actually not doing an A/B Test, it's just an example on the book. $\endgroup$ Nov 26, 2019 at 8:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.