I am reading mixed opinions comparing bayesian A/B testing and frequentist ones.

Basically my understanding is the following; consider the case of A/B testing with binary outcomes (https://www.evanmiller.org/bayesian-ab-testing.html#mjx-eqn-binary_ab_pr_alpha_b):

  • With frequentist approach, you basically calculate the $p-value$ from a contingency table, then you reject the null hypothesis that $A$ and $B$ are the same if the $p-value$ is smaller than the significance level $\sigma$, using a $\chi^2$ test. (https://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.stats.chi2_contingency.html).

  • With Bayesian approach, you fix a prior, you update it with the data of the contingency table, and then you come out with two probability distributions, that of $P_a$, the conversion rate in scenario $A$, and that of $P_B$. A typical choice of a prior is a Beta distribution, that remains a Beta after the update, but with updated parameters. For example, if I set the uniform prior, then after the update I will have that $$P_i \approx \beta(Success_i,Failures_i)$$ for $i=A,B$. At this point, I can calculate $Prob(P_a>P_B)$.

In the Bayesian approach, I can easily set the decision rule that I reject the null hypothesis if $$ Prob(P_a>P_B)>1-\sigma/2\;\;\;\;or\;\;\;\;Prob(P_a>P_B)<\sigma/2 $$ How will this test relate with the standard $\chi^2$ test?

I read many people saying that if we use uniform prior, frequentist and Bayesian are equivalent. This means that the test above would return the same results of a standard $\chi^2$ test?


1 Answer 1


Here are miscellaneous points, ending with the central question:

  1. It's important to realize that what marketers call an A-B test is what social scientists and statisticians more usually call an "experiment", or more fully, a "randomized experiment" or "true experiment" (or a study with "random assignment"); these are the terms you should search for if you're looking for information about these topics.

  2. Bayesian and frequentist methods each comprise broad approaches to data analysis and the interpretation of probability, rather than being a particular algorithim. A significance test like a $χ^2$-test of independence is but one frequentist approach to the analysis of an experiment.

  3. The test you propose for the Bayesian case doesn't make sense because a typical prior implies a probability of 0 that any two parameters are exactly equal. So if you gave each of $P_A$ and $P_B$ a beta prior and assumed prior independence, then the prior probability of the null hypothesis (viz., $P_A = P_B$) is 0. The posterior probability will be 0, too, no matter your data (in a phenomenon related to Cromwell's rule). This is one reason that significance tests aren't often conducted in a Bayesian fashion.

  4. To return to the idea that a uniform prior makes Bayesian and frequentist methods equivalent, here's how to make that precise. If a parameter has a uniform prior, then its (frequentist) maximum likelihood estimate is equal to its (Bayesian) posterior mode (also called the "maximum posterior probability (MAP) estimate").

  • $\begingroup$ I disagree with point 3. The test may be interpreted in a different way: I claim that Pa and Pb are "different enough", only if the test says that Pa > Pb or Pb>Pa with very high probability. When Prob(Pa > Pb) and Prob(Pb>Pa) are small, it means that the test is not able to establish which is bigger with high probability. So: I agree with the fact that in this case the null hypothesis shouldn't be Pa=Pb But the test still makes sense to spot "a difference between Pa and Pb that is observable with high probability; the null hypothesis in this case would be |Pa-Pb|<t for some threshold t. $\endgroup$ Commented Oct 7, 2017 at 14:23
  • $\begingroup$ @UlderiqueDemoitre I think you might have $|P_A - P_B|$ confused with the probability that $P_A > P_B$. The latter should converge to either 0 or 1 as you collect data. By contrast, the posterior distribution of $|P_A - P_B|$ can converge to any number in $[0, 1]$. $\endgroup$ Commented Oct 7, 2017 at 14:40
  • $\begingroup$ If $|P_A-P_B|$ is small, and I have not enough data, then both $Prob(P_A>P_B)$ and $Prob(P_A<P_B)$ will be far from 0 or 1 and I will not be able to claim anything with high probability. Increasing the sample size, the probabilities will converge to 1 or to 0, regardless $|P_A-P_B|$. Fixing the sample size, but increasing $|P_A-P_B|$, again the probabilities will converge to 1 or to 0. So, the scope of my test: Given the sample size, is $|P_A-P_B|$ big enough to say with high probability that $P_A>P_B$ or viceversa? $\endgroup$ Commented Oct 7, 2017 at 14:56
  • $\begingroup$ @UlderiqueDemoitre That's all correct, but the proposed "test" is not a significance test, and in particular doesn't do what the mentioned $χ^2$-test does. $\endgroup$ Commented Oct 7, 2017 at 15:50
  • $\begingroup$ @UlderiqueDemoitre If I answered your question to your satisfaction, you can accept my answer by clicking the check mark under the voting arrows. $\endgroup$ Commented Oct 11, 2017 at 15:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.