I'm performing some A/B split testing. Typically, I use a pairwise proportion test to determine if the difference between Control and Treatment(s) is statistically significant. I am researching Bayesian analysis on this topic. Specifically, I'm using bayes.prop.test from the Bayesian First Aid package in R.

As an example, a test I did compared Control with 2 Treatments. The chi-squared test returns a p-value of 0.351 so we fail to reject the null. The Bayesian analysis, though, will suggest that the Treatment 1 has a 87% and 90% probability of being less than the Control and Treatment 2, respectively. I would say that saying something has close to a 90% probability is significant and so I feel the Bayes test is more compelling.

My first question is do you agree with my thought process about the strength of the Bayesian inference? Secondly, should I expect this to always happen when performing both tests? I ask because if the answer is yes, it seems somewhat pointless to use both.


Your question is spot on. The frequentist approach is based on amassing evidence against an assumption that an effect is absent, when point null hypotheses are used. Bayesians generally disdain hypotheses (and Bayes factors) in favor of quantifying evidence for all possible levels of an effect. The most basic evidence if the posterior probability that the effect is in the right direction. Then you can move onto the probability that an effect is greator than $\epsilon$ for some $\epsilon > 0$. There are two reasons why the impressions you get from frequentist p-values and Bayesian posterior probabilities will differ:

  • Bayesian probabilities are directional, whereas most frequentist tests are two-sided so as to penalize you for possibly making a false claim that the effect is backwards (something that most investigators are not interested in)
  • Evidence for an effect given the data (Bayesian posterior) is different from evidence for extreme data given no effect (p-value)

For these reasons, Bayesian posterior distributions are more consistent with optimal decision making. If one were playing the odds and cost was not a factor, one would make money by betting on the effectiveness of a procedure if the posterior probability of it working in the right direction were > 0.5, given a reasonable prior distribution.


Disclaimer: I'm not to familiar with A/B testing.

One of the strengths of Bayesian statistics is the use of priors, hyperpriors, and the hierarchy. Right now you have a flat beta(1,1) prior on all your outcomes. There must be some prior information you can include.

I am not entirely sure what you mean it has "a 87% and 90% probability of being less than the Control and Treatment 2, respectively". Are you not looking at 95% credible difference in group estimates?

And to answer your questions: yes, the Bayesian approach can definitely be stronger. It all depends on the data and prior, though. If you use a flat prior, there is not much benefit to the Bayesian approach (IMO). Second, again it depends on the prior. With a flat prior you would probably see an agreement most of the time.

To finish, I would really recommend reading Doing Bayesian Data Analysis. Not the whole book, but just the first few chapters, to get a better intuition on Bayesian statistics.

  • $\begingroup$ The bayes.prop.test function in R provides both a credible interval for the differences and the percentages of the distribution that lie above or below zero. So when I say 87% probability less than control, I'm referencing that 87% of the distribution for 'Treatment 1 minus Control' lies below zero. $\endgroup$ – c.custer Oct 26 '16 at 17:44
  • $\begingroup$ I see. I thought you had only included the standard output of the function. Nonetheless, I think it's an important point of Bayesian statistics that you touch upon. There are no pre-defined cutoffs for "significance". Do you think that 87% (or 90%) probability is significant? Can you convince yourself about this? And also, how do prior choices influence your outcome? Just to add: did you run convergence statistics on your model? $\endgroup$ – demodw Oct 26 '16 at 19:04
  • $\begingroup$ @demodw you are confusing credible intervals with posterior probabilities $\endgroup$ – Frank Harrell May 10 '19 at 14:48

What you are reporting is not the result of a Bayesian test. First off, from a very high level, proper Bayesian testing is done with Bayes Factors. Furthermore, you must prespecify (prior to even looking at the data) the objectives of the analysis including the comparisons, ways to control for FWER or FDR if considering more than one, and clinically relevant thresholds of effect and evidence.

What you describe as the "Bayesian test" is a descriptive analysis with a few problems. It is not inference. You are simply describing the posterior distribution of the sample proportion. You specially picked Treatment 2 to compare to Treatment 1 and Control because it had a notable mean difference. Secondly, unless it's a matched design (A/B tests as a general concept can be done many ways), one has to ask if the 87% and 90% you calculate are based on the posterior of Treatment 1 and a point estimate of Treatment 2 and control. Lastly, the choice of thresholding your Bayesian belief (of 87% and 90%) into a "significant" vs "non-significant" designation is totally arbitrary. Whether you think it's significant or not is arbitrary because you've already done the data analysis. In each case, you did not prespecify what the objectives of the analysis were, and it doesn't matter whether one is Frequentist or Bayesian when they do that, the inference is invalid.


My first question is do you agree with my thought process about the strength of the Bayesian inference?

No. I do not agree. You should not run both together and which one you run should be based on the problem that you are seeking to solve. Also, avoid Bayes factors until you are very skilled. When you have non-binary choices, they have all of the issues that p-values have. Consider reading:

Michael Lavine and Mark J Schervish (May 1999). Bayes Factors: What They Are and What They Are Not. The American Statistician. 53(2). pp. 119-122

When used as intended, they are fine but be careful not to infer too much from them. Likewise, you should read:

Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E.-J. (2011). Statistical evidence in experimental psychology: An empirical comparison using 855 t tests. Perspectives on Psychological Science, 6, 291-298.

You should use a Bayesian method in four primary circumstances.

The first one is when you have prior information regarding the relative probability of which of the two models are true. If you have prior knowledge from research or experience, then it should be quantified into the prior and you should be using the posterior odds ratio rather than Bayes factors.

The second one is when you are gambling money on the outcome, such as budgeting resources, setting inventory or so forth.

The third one is when your primary concern is to estimate the value of a parameter accurately, but you are unconcerned with bias. If you are willing to exchange bias for accuracy, then you should use Bayes.

The fourth one is when you want to assess the subjective probabilities of two or more sets of cases or parameters. So, for example, there is a 90% chance model X is true, a 6% chance that model Y is true and a 4% chance model Z is true. It is valuable for both model selection and model averaging. It is an excellent exploratory tool.

There are five reasons to use Frequentist methods.

You should use Frequentist methods when you lack prior information and, since you have no prior information to condition on to protect you from statistical runs and strange results, you want to minimize the maximum possible loss you could experience from being unlucky and choosing a bad sample.

You should also use Frequentist methods when you want to remove broad classes of potential hypotheses from consideration conclusively. Fisher's no effect hypothesis is an incredibly powerful tool to cut through the crap in science when it is used as intended. It can foreclose many competing hypothesis by assuming the one you believe to be true to be false. Falsification of the false statement is very close to accepting the true statements.

Although unbiased estimators are informationally costly, they have the tremendous advantage that they are unbiased. As a tool for rhetoric, an argument built around an unbiased estimator avoids the inevitable discussion of whether the bias introduced by the estimation method caused the results to be different in a material way from another method.

The fourth reason is in the case of a sharp null hypothesis, such as when the hypothesis is $\theta=k,\text{ } \theta\in\Theta$ where $\Theta$ is continuous. Bayesian methods have no good way to solve this issue. There are workarounds such as ROPE, regions of practical equivalence, but they are a bit dangerous because they really must be equivalent. Consider $\Pr(\theta|X)\sim\mathcal{N}(0,1)$ and a sharp null of $\theta=0$. It is true that $\theta$ is near zero if $\pm{3}$ is considered close, but if it is not, then while it is true that zero is in the highest density region, so is -1 and 1. There are an infinite number of possible values in the highest density region, only one of them is zero.

Remember the difference is that the Frequentist test would be $\Pr(X|\theta=0)$ versus the Bayesian test $\Pr(\theta=0|X)$. If $\theta$ is continuous, then $\Pr(\theta=0|X)=0.$

Finally, if there is a material dispute regarding the prior distribution in the field, a Frequentist estimator avoids this by ignoring the prior distribution.

Secondly, should I expect this to always happen when performing both tests?

You shouldn't be performing both sets of tests. They are built on conflicting assumptions. The posterior probabilities and the p-value are orthogonal constructions. They should not be interpreted similarly.

For example, you chose to use the word significant with a posterior probability above. When speaking of Bayesian probabilities, you should drop the phraseology and the thinking of p-values.

Imagine you saw 10,000,000 observations and P(X)=.9, P(Y)=.1, is that significant? Arguably it is not. If you saw 10,000,000 of something and do not have conclusive evidence such as P(X)=.999999, then maybe you should be concerned. On the other hand, if you saw 10 observations and P(X)=.9 then you should be very excited because that is a strong probability for such a small sample. You should not be thinking in terms of an $\alpha$ cut-off value. These are straight probabilities that something is true.

Likewise, the null is a very arrogant and powerful statement. If your null is $\mu=0$ then you are saying there is a 100% chance that $\mu$ is 0 and then you are looking at how extreme your data would have to be if the null were true. Is the deductive reasoning available in Frequentist methods useful?

Your choice between accepting the arrogance of creating a prior distribution and the arrogance of the null hypothesis should be based on what your needs are.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.