I am reading mixed opinions comparing bayesian A/B testing and frequentist ones.
Basically my understanding is the following; consider the case of A/B testing with binary outcomes (https://www.evanmiller.org/bayesian-ab-testing.html#mjx-eqn-binary_ab_pr_alpha_b):
With frequentist approach, you basically calculate the $p-value$ from a contingency table, then you reject the null hypothesis that $A$ and $B$ are the same if the $p-value$ is smaller than the significance level $\sigma$, using a $\chi^2$ test. (https://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.stats.chi2_contingency.html).
With Bayesian approach, you fix a prior, you update it with the data of the contingency table, and then you come out with two probability distributions, that of $P_a$, the conversion rate in scenario $A$, and that of $P_B$. A typical choice of a prior is a Beta distribution, that remains a Beta after the update, but with updated parameters. For example, if I set the uniform prior, then after the update I will have that $$P_i \approx \beta(Success_i,Failures_i)$$ for $i=A,B$. At this point, I can calculate $Prob(P_a>P_B)$.
In the Bayesian approach, I can easily set the decision rule that I reject the null hypothesis if $$ Prob(P_a>P_B)>1-\sigma/2\;\;\;\;or\;\;\;\;Prob(P_a>P_B)<\sigma/2 $$ How will this test relate with the standard $\chi^2$ test?
I read many people saying that if we use uniform prior, frequentist and Bayesian are equivalent. This means that the test above would return the same results of a standard $\chi^2$ test?