# Bayesian AB Testing: Simulated vs analytically derived posterior

Consider the problem of distributing two leaflets A and B to increase sales of a given product. We distribute 20 of leaflet A (with 8 successes) and 20 of leaflet B (with 3 successes). We know want to know what the probability is that leaflet A is better than leaflet B.

We can do this in two ways, the first is simulation. Using for example a uniform distribution as a prior which we sample our binomial parameter p from. Or alternatively, through utilising the fact that the beta distribution is a conjugate prior of the binomial. As such we can just perturb our initial prior configuration (likely to be Beta(1,1) as it is the same a uniform distribution) by the data to obtain our posterior.

If i want to now calculate "the probability is that leaflet A is better than leaflet B"

I have read here that i can do this by:

First, generate N pairs of random samples from the joint distribution. Since they are independent, you need only pick the first item in the pair from the first beta distribution and similarly for the second item. You may need to generate a lot of samples, if the two rates are close to each other. Count how many of the pairs have the first number greater than the second. Let us call this number k. Then you can get the estimate for the probability P(r₁ > r₂) simply dividing k / N.

This makes sense for the analytically derived example but can i extend it to the simulated example.

In other words can I randomly draw pairs of values from the two simulated distributions for A and B, compare their values (repeat a few thousand times) and the resulting tally will be the answer to my questions?

Thanks in advance, happy to clarify if needed

Edit: The simulated case (sampling the parameter of the binomial distribution (p) from a uniform prior):

The two leaflets are independent, so we can examine their posteriors independently.

Leaflet A has a posterior of

$$p_A \sim \operatorname{Beta}(8+1, 12+1)$$

Leaflet B has a posterior of

$$p_B \sim \operatorname{Beta}(3+1, 17+1)$$

We can compute the probability that $$p_A>p_B$$ by just simulating from these distributions and comparing element by element if our draw from A's posterior is larger than the draw from B's posterior.

set.seed(0)
pa = rbeta(1000, 9, 13)
pb = rbeta(1000, 4, 18)

mean(pa>pb)
>>>0.953


This means that 95.3% of the draws from A's posterior were larger than the draws from B's posterior. This, we estimate $$P(p_A>p_B) = 0.953$$.

• Thanks for your response. I realise that but what if instead of using the fact that the beta is conjugate to the binomial I used simulation? Could i just randomly draw pairs and compare them in the same way as is done for the conjugate case? Mar 5, 2020 at 17:30
• I have added what the simulated case looks like (I have done the conjugate case also - the distributions look largely identical). Mar 5, 2020 at 17:32