Are p-value and Bayesian probabilities equivalent when applied to conversion rates? For evaluating AB-tests, one can use the frequentist approach or a Bayesian one.
Using these websites I compared both with some example data:

*

*Bayesian A/B-test Calculator

*AB Testguide
Result:
+---------+---------------+---------+---------------+---------+--------------------------------------+
| Users A | Conversions A | Users B | Conversions B | p-value | Bayesian chance of A outperforming B |
+---------+---------------+---------+---------------+---------+--------------------------------------+
| 1000    | 100           | 1000    | 120           | 0.0764  | 7.7%                                 |
| 1000    | 100           | 1000    | 130           | 0.0176  | 1.8%                                 |
| 10000   | 1000          | 10000   | 1100          | 0.0105  | 1.1%                                 |
| 10000   | 1000          | 10000   | 1030          | 0.2412  | 24.1%                                |
| 17000   | 1700          | 10000   | 1030          | 0.2157  | 21.5%                                |
| 4000    | 2683          | 3000    | 2000          | 0.6402  | 64.1%                                |
+---------+---------------+---------+---------------+---------+--------------------------------------+

I'm aware of the different meanings of the probabilities given by the p-value and by the Bayesian chance of A outperforming B, thus it surprises me, that both seem to give the same values.
So are those two really equivalent?
And if so, why do all the simulation efforts involved in the Bayesian calculation?
 A: Notice that the last three cases don't give exactly equal answers when properly rounded (e.g. in the last one, the t-test gives 64.0% vs 64.1% in Bayes).
If you have two iid. normally distributed groups of equal size, then the p-value is going to be equal to the Bayesian result because they are the same function. The only difference lies in the interpretation.
The Bayesian calculator gives you extra knobs to answer additional questions in a business-user-friendly way. I'm not sure if the traffic/test duration variable is actually used to tune the precision, but if they do, this is another trick that approach buys you.
Finally, t-tests depend on a set of assumptions. There is some leeway, but at the very least you're shackled to a unimodal Gaussian. The online calculator doesn't support it, but the Bayesian approach is much, much more flexible, and you can use the underlying framework for all kinds of crazy distributions you may see in the real world, and to pool the results of several A/B tests in a principled way.
