Encoding prior knowledge into an AB test

Question

Let's say we are running an A/B test on our website which has only American and French visitors (assuming both groups constitute about half of the visitor population).

Although it is not a good practice, we divide visitors based on their country and show them two different banners. The ultimate goal is to pick the banner with a higher click-through rate (CTR).

         success failure    CTR
banner_A    1850   31400  0.056  --> French visitors
banner_B    1950   32000  0.057  --> American visitors

One way to test the statistical significance is to use only these numbers and conduct Pearson's chi-squared test. In this case p-value seems to be 0.321; and the higher CTR of banner_B seems statistically insignificant.

However, let's also assume that 51% of the past visitors who clicked this sort of banners were French visitors (see below). So, the priors on CTRs were not equal in the first place.

            success failure    CTR
banner_Past    5100   80000  0.060  --> French visitors
banner_Past    4900   80200  0.058  --> American visitors

How can we encode this "51% tendency" kind of prior information into our statistical significance test?

Answer 1

Update: I misunderstood what was the nature of the prior information. You have a problem with your experiment, because we can't disentangle the effect of the nationality from the effect of the banner. As far as I understood, banner_past is different from banner A and B. So, you know that, on banner-past, french are slightly more likely to click on the banner. Now, since you didn't show banner A to any american, and banner B to any french, how can we estimate the effect of each banner, conditional on the nationality? It can't be done, as far as I can imagine.

Old Answer: Use Bayes!

For simplicity here, I'll assume the probability of clicking on banner $A$ follows a Binomial distribution with probability $\theta_{a}$ and the probability of clicking on banner $B$ follows a Binomial distribution with probability $\theta_{b}$. You're interested on knowing if $\theta_{a}$ > $\theta_{b}$. If so, you`ll use banner A.

If we assume a conjugated prior for this likelihood, namely, a beta distribution for $\theta_{a}$ with paramaters ($\alpha_{a}$, $\beta_{a}$) and another one with parameters ($\alpha_{b}$, $\beta_{b}$) for $\theta_{b}$, where $\alpha_{a}$ is the number of clicks on banner $A$ and $\beta_{a}$ is the number of people that saw banner $A$ but didn't click on it. The same for banner $B$. You can use prior information on the number of clicks on banner $A$ and on the number of clicks on banner $B$ and compute the posterior probability of $\theta_{a}$ and of $\theta_{b}$. The posteriori will be Beta distributions with parameters ($\alpha_{a}$ + number of clicks on $A$ on the new experiment, $\beta_{a}$ + number of people who saw banner $A$, but didn't click on it on the new experiment) and ($\alpha_{b}$ + number of clicks on $B$ on the new experiment, $\beta_{b}$ + number of people who saw banner $B$, but didn't click on it on the new experiment) for banners $A$ and $B$ respectively. To compute the posterior probability of $\theta_{a}$ - $\theta_{b}$, just run a simulation.

Please note that in this modeling I assumed that you have prior information not only on the 51%, but also on the number of cases that resulted in this 51% number. This is important to weight how much information this 51% really carries on. 51% with say 100 cases is much less informative than 10,000 cases.

Update: Based on your commnet, here's a R code to your question:

 set.seed(2)
 posterioriA <- rbeta(n_trials, 5100 + 1850, 80000-5100 + 31400-1850 )
 posteriorib <- rbeta(n_trials, 4900 + 1950, 80200-4900 + 32000-1950 )
 posterioriDifAB <- posterioriA - posteriorib

 # probability that banner A is better than Banner B, i.e., it has a higher probability of click
 sum(posterioriDifAB > 0)/length(posterioriDifAB)
 # in my computer, 0.9063

So, the posterior probability of Banner A being better than banner B is 90%. You can compute, also, the probability that banner A is substantially better, if you're able to define this mathematically. Say, for instance, that a substantive improvement is defined as banner A having a probability of click 0.1 p.p bigger than banner B. To see if this is the case, just compute:

 sum(posterioriDifAB > 0.001)/length(posterioriDifAB)
 # in my computer, 0.6335

So, based on this, although Banner A is statistically better than banner B, it's not substantially better.