# Applying Bayesian statistics to A/B testing, calculate credible intervals?

I do not know much about statistics but from my primitive research, I would like to explore how to apply Bayesian statistics in A/B testing.

The best Bayesian-based A/B split test graphic calculator I have encountered so far calculates the "Apprx. probability of being best", and uses a simulation with jStats to determine 95% confidence intervals.

So far, I can calculate the "Apprx. probability of being best" column in the calculator above with probability_B_beats_A() for A/B and probability_C_beats_A_and_B(). I am not sure how to calculate more than 3 variations but I can live with that.

How about credible intervals? possibility without resorting to running a simulation?

I found this function that seemingly calculates what I need from Frequentism and Bayesianism III: Confidence, Credibility, and why Frequentism and Science do not Mix, but I have no idea how to use:

from scipy.special import erfinv

def freq_CI_mu(D, sigma, frac=0.95):
"""Compute the confidence interval on the mean"""
# we'll compute Nsigma from the desired percentage
Nsigma = np.sqrt(2) * erfinv(frac)
mu = D.mean()
sigma_mu = sigma * D.size ** -0.5
return mu - Nsigma * sigma_mu, mu + Nsigma * sigma_mu

print("95% Confidence Interval: [{0:.0f}, {1:.0f}]".format(*freq_CI_mu(D, 10)))


What are D & sigma? Thanks for bearing with my primitive understanding of Bayesian statistics.

• Note that a CI is only as useful as the normal approximation is useful. If the statistic you are interested has a highly skewed distribution or has a truncated support the CI may lead to poor interpretations of the nature of the estimator. For Credible Intervals (or more generally sets), the normality approximation isn't important, but the credible set may be constituted of lots of little subsets of the support for a given $\alpha$ level. Calculating the credible interval or set is also difficult for anything but well behaved posterior distributions, so numerical integration is likely. Commented Jun 2, 2016 at 20:17

For any set of $N$ values $D=\{x_i\}^N_{i=1}$, an unbiased estimate of the mean $\mu$ of the distribution is (...)
Note here that we've assumed $\sigma_x$ is a known quantity; this could also be estimated from the data along with $\mu$, but here we kept things simple for sake of example.
$D$ is vector of observations and $\sigma_x$ is standard deviation that is known in advance, or can be estimated.