Let's say we are running an A/B test on our website which has only American and French visitors (assuming both groups constitute about half of the visitor population).

Although it is not a good practice, we divide visitors based on their country and show them two different banners. The ultimate goal is to pick the banner with a higher click-through rate (CTR).

         success failure    CTR
banner_A    1850   31400  0.056  --> French visitors
banner_B    1950   32000  0.057  --> American visitors

One way to test the statistical significance is to use only these numbers and conduct Pearson's chi-squared test. In this case p-value seems to be 0.321; and the higher CTR of banner_B seems statistically insignificant.

However, let's also assume that 51% of the past visitors who clicked this sort of banners were French visitors (see below). So, the priors on CTRs were not equal in the first place.

            success failure    CTR
banner_Past    5100   80000  0.060  --> French visitors
banner_Past    4900   80200  0.058  --> American visitors

How can we encode this "51% tendency" kind of prior information into our statistical significance test?


1 Answer 1


Update: I misunderstood what was the nature of the prior information. You have a problem with your experiment, because we can't disentangle the effect of the nationality from the effect of the banner. As far as I understood, banner_past is different from banner A and B. So, you know that, on banner-past, french are slightly more likely to click on the banner. Now, since you didn't show banner A to any american, and banner B to any french, how can we estimate the effect of each banner, conditional on the nationality? It can't be done, as far as I can imagine.

Old Answer: Use Bayes!

For simplicity here, I'll assume the probability of clicking on banner $A$ follows a Binomial distribution with probability $\theta_{a}$ and the probability of clicking on banner $B$ follows a Binomial distribution with probability $\theta_{b}$. You're interested on knowing if $\theta_{a}$ > $\theta_{b}$. If so, you`ll use banner A.

If we assume a conjugated prior for this likelihood, namely, a beta distribution for $\theta_{a}$ with paramaters ($\alpha_{a}$, $\beta_{a}$) and another one with parameters ($\alpha_{b}$, $\beta_{b}$) for $\theta_{b}$, where $\alpha_{a}$ is the number of clicks on banner $A$ and $\beta_{a}$ is the number of people that saw banner $A$ but didn't click on it. The same for banner $B$. You can use prior information on the number of clicks on banner $A$ and on the number of clicks on banner $B$ and compute the posterior probability of $\theta_{a}$ and of $\theta_{b}$. The posteriori will be Beta distributions with parameters ($\alpha_{a}$ + number of clicks on $A$ on the new experiment, $\beta_{a}$ + number of people who saw banner $A$, but didn't click on it on the new experiment) and ($\alpha_{b}$ + number of clicks on $B$ on the new experiment, $\beta_{b}$ + number of people who saw banner $B$, but didn't click on it on the new experiment) for banners $A$ and $B$ respectively. To compute the posterior probability of $\theta_{a}$ - $\theta_{b}$, just run a simulation.

Please note that in this modeling I assumed that you have prior information not only on the 51%, but also on the number of cases that resulted in this 51% number. This is important to weight how much information this 51% really carries on. 51% with say 100 cases is much less informative than 10,000 cases.

Update: Based on your commnet, here's a R code to your question:

 posterioriA <- rbeta(n_trials, 5100 + 1850, 80000-5100 + 31400-1850 )
 posteriorib <- rbeta(n_trials, 4900 + 1950, 80200-4900 + 32000-1950 )
 posterioriDifAB <- posterioriA - posteriorib

 # probability that banner A is better than Banner B, i.e., it has a higher probability of click
 sum(posterioriDifAB > 0)/length(posterioriDifAB)
 # in my computer, 0.9063

So, the posterior probability of Banner A being better than banner B is 90%. You can compute, also, the probability that banner A is substantially better, if you're able to define this mathematically. Say, for instance, that a substantive improvement is defined as banner A having a probability of click 0.1 p.p bigger than banner B. To see if this is the case, just compute:

 sum(posterioriDifAB > 0.001)/length(posterioriDifAB)
 # in my computer, 0.6335

So, based on this, although Banner A is statistically better than banner B, it's not substantially better.

  • $\begingroup$ I did a simple Monte Carlo simulation: n_trials <- 10000 A_samples <- rbinom(n_trials, views_A, clicks_A/views_A) * rbeta(n_trials, clicks_A_past, fails_A_past) B_samples <- rbinom(n_trials, views_B, clicks_B/views_B) * rbeta(n_trials, clicks_B_past, fails_B_past) (mean(A_samples < B_samples)) Is my understanding correct? $\endgroup$
    – srctaha
    Nov 17, 2013 at 5:19
  • $\begingroup$ I was expecting something different from the Bayesian formulation. The experiment shows us that CTR B is higher than CTR A, despite the fact that visitors of banner A have a higher tendency of clicking whatever we show them. So I was thinking that the prior contributes to the reliability of banner B, but this formulation concludes that "posterior probability of banner A being better than banner B is 90%"!? $\endgroup$
    – srctaha
    Nov 20, 2013 at 8:24
  • $\begingroup$ I may have misunderstood your question. I thought your prior information was that Banner A was better than banner B. But what you're saying is that you have no prior about banner A or B, but only that french visitors were more likely to click on the first place. I'll update my answer to reflect this fact. $\endgroup$ Nov 28, 2013 at 12:10
  • $\begingroup$ This was a purely hypothetical question. What I wanted to indicate by banner_Past is that one of the two groups under consideration has a different initial response ratio than the other; let that response be clicking banners or making a purchase after a sales pitch. $\endgroup$
    – srctaha
    Dec 11, 2013 at 1:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.