AB testing vs testing the null hypothesis

Question

I'm trying to understand the difference between

1. testing the null hypothesis (i.e. testing that the probability of a "goal" is the same across 2 different populations, similar to prop.test in R)

2. an A/B test using a bayesian formula such as described here: http://www.evanmiller.org/bayesian-ab-testing.html

Is there a difference? Is one preferable?

The problem I'm facing looks something like this:

control group has 100,000 impressions and 100 reactions test group has 50,000 impressions and 55 reactions

Answer 1

Is there a difference?

Yes. A null hypothesis test produces a test statistic and a p-value, the probability of a test statistic as extreme as the that of the data, under the assumption that the null hypothesis is true. In your example, prop.test tests the assumption that the $p_A$ and $p_B$ are equal. This is distinct from the probability described in your link, $Pr(p_B \gt p_A)$:

On your data, prop.test produces a p-value of 0.6291; we interpret this to mean that if $p_A = p_B$, we would expect to see data this extreme in roughly 63% of experiments. But this isn't directly interpretable as the probability that the alternative outperforms the control. Using the linked post's formula, one arrives at $Pr(p_B \gt p_A) \approx 0.726$, which is directly interpretable as such. (Python code after the break.)

To gain a little intuition about this, observe the two posterior densities for $p_A, p_B$.

1. The mode of $p_B$ is clearly to the right of the mode of $p_A$. In other words, our point estimate for $p_B$ is higher. Expected, since $\frac{55}{50000} \gt \frac{100}{100000}$.
2. The posterior for $p_B$ is more dispersed. Intuitively satisfying: since we've observed A twice as many times, we're more confident in a narrower posterior.
3. There's still plenty of overlap—it's conceivable that the two treatments just don't meaningfully differ.

For one last intuitive aid, we can plot the distribution of the difference of the posteriors, and observe that roughly three-quarters of its area lies to the right of $0$:

To reiterate, the p-value only tells us that the data fail to reach the extremity at which we'd be convinced a difference exists.

Is one preferable?

That question is an instance of the broader Bayesian v. Frequentist choice, and often veers into matters of opinion. In general, I believe the answer depends on many factors, including application, audience, and analyst preference. Here are a few ways to view the difference between the two, which will hopefully help show when one might be preferable.

One nice introduction to Bayesian A/B testing puts it like so:

Which of these two statements is more appealing:

(1) "We rejected the null hypothesis that A=B with a p-value of 0.043."

(2) "There is an 85% chance that A has a 5% lift over B."

Bayesian modeling can answer questions like (2) directly.

For another take, theoretical statistician Larry Wasserman nicely describes the two schools of thought:

But first, I should say that Bayesian and Frequentist inference are defined by their goals not their methods.

The Goal of Frequentist Inference: Construct procedure with frequency guarantees. (For example, confidence intervals.)

The Goal of Bayesian Inference: Quantify and manipulate your degrees of beliefs. In other words, Bayesian inference is the Analysis of Beliefs.

---

>>> from scipy.special import betaln as lbeta
def probability_B_beats_A(a_A, b_A, a_B, b_B):
...     total = 0.0
...     for i in range(a_B):
...         total += exp(lbeta(a_A+i, b_B+b_A) - log(b_B+i) - lbeta(1+i, b_B) - lbeta(a_A, b_A))
...     return total
>>> probability_B_beats_A(101, 100001 - 100, 56, 50001 - 55)
0.72594700264280843