# Encoding prior knowledge into an AB test

Let's say we are running an A/B test on our website which has only American and French visitors (assuming both groups constitute about half of the visitor population).

Although it is not a good practice, we divide visitors based on their country and show them two different banners. The ultimate goal is to pick the banner with a higher click-through rate (CTR).

         success failure    CTR
banner_A    1850   31400  0.056  --> French visitors
banner_B    1950   32000  0.057  --> American visitors


One way to test the statistical significance is to use only these numbers and conduct Pearson's chi-squared test. In this case p-value seems to be 0.321; and the higher CTR of banner_B seems statistically insignificant.

However, let's also assume that 51% of the past visitors who clicked this sort of banners were French visitors (see below). So, the priors on CTRs were not equal in the first place.

            success failure    CTR
banner_Past    5100   80000  0.060  --> French visitors
banner_Past    4900   80200  0.058  --> American visitors


How can we encode this "51% tendency" kind of prior information into our statistical significance test?

Update: I misunderstood what was the nature of the prior information. You have a problem with your experiment, because we can't disentangle the effect of the nationality from the effect of the banner. As far as I understood, banner_past is different from banner A and B. So, you know that, on banner-past, french are slightly more likely to click on the banner. Now, since you didn't show banner A to any american, and banner B to any french, how can we estimate the effect of each banner, conditional on the nationality? It can't be done, as far as I can imagine.

For simplicity here, I'll assume the probability of clicking on banner $A$ follows a Binomial distribution with probability $\theta_{a}$ and the probability of clicking on banner $B$ follows a Binomial distribution with probability $\theta_{b}$. You're interested on knowing if $\theta_{a}$ > $\theta_{b}$. If so, youll use banner A.

If we assume a conjugated prior for this likelihood, namely, a beta distribution for $\theta_{a}$ with paramaters ($\alpha_{a}$, $\beta_{a}$) and another one with parameters ($\alpha_{b}$, $\beta_{b}$) for $\theta_{b}$, where $\alpha_{a}$ is the number of clicks on banner $A$ and $\beta_{a}$ is the number of people that saw banner $A$ but didn't click on it. The same for banner $B$. You can use prior information on the number of clicks on banner $A$ and on the number of clicks on banner $B$ and compute the posterior probability of $\theta_{a}$ and of $\theta_{b}$. The posteriori will be Beta distributions with parameters ($\alpha_{a}$ + number of clicks on $A$ on the new experiment, $\beta_{a}$ + number of people who saw banner $A$, but didn't click on it on the new experiment) and ($\alpha_{b}$ + number of clicks on $B$ on the new experiment, $\beta_{b}$ + number of people who saw banner $B$, but didn't click on it on the new experiment) for banners $A$ and $B$ respectively. To compute the posterior probability of $\theta_{a}$ - $\theta_{b}$, just run a simulation.

Please note that in this modeling I assumed that you have prior information not only on the 51%, but also on the number of cases that resulted in this 51% number. This is important to weight how much information this 51% really carries on. 51% with say 100 cases is much less informative than 10,000 cases.

Update: Based on your commnet, here's a R code to your question:

 set.seed(2)
posterioriA <- rbeta(n_trials, 5100 + 1850, 80000-5100 + 31400-1850 )
posteriorib <- rbeta(n_trials, 4900 + 1950, 80200-4900 + 32000-1950 )
posterioriDifAB <- posterioriA - posteriorib

# probability that banner A is better than Banner B, i.e., it has a higher probability of click
sum(posterioriDifAB > 0)/length(posterioriDifAB)
# in my computer, 0.9063


So, the posterior probability of Banner A being better than banner B is 90%. You can compute, also, the probability that banner A is substantially better, if you're able to define this mathematically. Say, for instance, that a substantive improvement is defined as banner A having a probability of click 0.1 p.p bigger than banner B. To see if this is the case, just compute:

 sum(posterioriDifAB > 0.001)/length(posterioriDifAB)
# in my computer, 0.6335


So, based on this, although Banner A is statistically better than banner B, it's not substantially better.

• I did a simple Monte Carlo simulation: n_trials <- 10000 A_samples <- rbinom(n_trials, views_A, clicks_A/views_A) * rbeta(n_trials, clicks_A_past, fails_A_past) B_samples <- rbinom(n_trials, views_B, clicks_B/views_B) * rbeta(n_trials, clicks_B_past, fails_B_past) (mean(A_samples < B_samples))` Is my understanding correct? Nov 17, 2013 at 5:19
• I was expecting something different from the Bayesian formulation. The experiment shows us that CTR B is higher than CTR A, despite the fact that visitors of banner A have a higher tendency of clicking whatever we show them. So I was thinking that the prior contributes to the reliability of banner B, but this formulation concludes that "posterior probability of banner A being better than banner B is 90%"!? Nov 20, 2013 at 8:24
• I may have misunderstood your question. I thought your prior information was that Banner A was better than banner B. But what you're saying is that you have no prior about banner A or B, but only that french visitors were more likely to click on the first place. I'll update my answer to reflect this fact. Nov 28, 2013 at 12:10
• This was a purely hypothetical question. What I wanted to indicate by banner_Past is that one of the two groups under consideration has a different initial response ratio than the other; let that response be clicking banners or making a purchase after a sales pitch. Dec 11, 2013 at 1:36