Bayesian vs frequentist A/B testing: in which sense they are equivalent with uniform priors?

Question

I am reading mixed opinions comparing bayesian A/B testing and frequentist ones.

Basically my understanding is the following; consider the case of A/B testing with binary outcomes (https://www.evanmiller.org/bayesian-ab-testing.html#mjx-eqn-binary_ab_pr_alpha_b):

• With frequentist approach, you basically calculate the $p-value$ from a contingency table, then you reject the null hypothesis that $A$ and $B$ are the same if the $p-value$ is smaller than the significance level $\sigma$, using a $\chi^2$ test. (https://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.stats.chi2_contingency.html).

• With Bayesian approach, you fix a prior, you update it with the data of the contingency table, and then you come out with two probability distributions, that of $P_a$, the conversion rate in scenario $A$, and that of $P_B$. A typical choice of a prior is a Beta distribution, that remains a Beta after the update, but with updated parameters. For example, if I set the uniform prior, then after the update I will have that $$P_i \approx \beta(Success_i,Failures_i)$$ for $i=A,B$. At this point, I can calculate $Prob(P_a>P_B)$.

In the Bayesian approach, I can easily set the decision rule that I reject the null hypothesis if $$ Prob(P_a>P_B)>1-\sigma/2\;\;\;\;or\;\;\;\;Prob(P_a>P_B)<\sigma/2 $$ How will this test relate with the standard $\chi^2$ test?

I read many people saying that if we use uniform prior, frequentist and Bayesian are equivalent. This means that the test above would return the same results of a standard $\chi^2$ test?

Answer 1

Here are miscellaneous points, ending with the central question:

1. It's important to realize that what marketers call an A-B test is what social scientists and statisticians more usually call an "experiment", or more fully, a "randomized experiment" or "true experiment" (or a study with "random assignment"); these are the terms you should search for if you're looking for information about these topics.

2. Bayesian and frequentist methods each comprise broad approaches to data analysis and the interpretation of probability, rather than being a particular algorithim. A significance test like a $χ^2$-test of independence is but one frequentist approach to the analysis of an experiment.

3. The test you propose for the Bayesian case doesn't make sense because a typical prior implies a probability of 0 that any two parameters are exactly equal. So if you gave each of $P_A$ and $P_B$ a beta prior and assumed prior independence, then the prior probability of the null hypothesis (viz., $P_A = P_B$) is 0. The posterior probability will be 0, too, no matter your data (in a phenomenon related to Cromwell's rule). This is one reason that significance tests aren't often conducted in a Bayesian fashion.

4. To return to the idea that a uniform prior makes Bayesian and frequentist methods equivalent, here's how to make that precise. If a parameter has a uniform prior, then its (frequentist) maximum likelihood estimate is equal to its (Bayesian) posterior mode (also called the "maximum posterior probability (MAP) estimate").