Suppose I've run an A/B/C experiment (same as A/B but with 3 groups instead of 2) and gathered the following data for number of participants in each group and number of desired actions in each group (e.g. clicks on a certain button):

\begin{array}{c|c|c|c} & a & b & c \\ \hline total & 1000 & 1100 & 1070 \\ clicks & 120 & 150 & 180 \end{array}

Conversion estimates are different for each group:

\begin{array}{c|c|c|c} conv & 0.120 & 0.136 & 0.168 \end{array}

How do I show the difference is statistically significant and select the best variant?

In A/B test with only two groups it is possible to compute distance between conversions and confidence interval using an equation

$$ conv_2 - conv_1 \pm t * \sqrt{ \frac{conv_1 (1 - conv_1)}{N_1} + \frac{conv_2 (1 - conv_2)}{N_2} } $$

where $t$-value is determined by desired confidence level ($t=1.96$ for $\alpha = 95 \%$ ). If the interval doesn't contain zero, then it is possible to select the version with the largest conversion, if the interval contains zero, then it is not possible to claim there is statistically significant difference between the two variants.

Is it still possible to perform pairwise comparison of A/B/C conversions using the equation above, but with $t$-value adjusted for multiple comparisons?

One possible adjustment is Bonferroni correction, where $t$-value for $(1 - (1 - \alpha) / m, \alpha = 0.95, m = 3 )$ confidence level should be used. This method is safe, but conservative.

Another method is Tukey's HSD where $t$-value should be replaced by $q$-value (e.g. from http://www.real-statistics.com/statistics-tables/studentized-range-q-table/ ). This is preferred over Bonferroni test.

So, what is a correct procedure to determine the best A/B/C-variant?

  • $\begingroup$ Tukey*. And this is the common method for comparing groups, an ANOVA type regression followed by a post-hoc analysis for individual differences (Tukey's HSD). However, clicks out of total is a ratio, which follows a binomial distribution. So you may want to use a binomial GLM instead. $\endgroup$ Jul 6 '19 at 8:38
  • $\begingroup$ "Tukey*" Ok, thanks! Is "ANOVA type regression" the same thing as ANOVA that computes F-value or am I missing something? If so, is it really necessary to perform ANOVA before Tukey's HSD? If Tukey's HSD finds an indistinguishable pair of values, doesn't that imply that F-score from ANOVA (that takes all variants into account) will be small enough and it won't be possible to discard null hypothesis? $\endgroup$ Jul 6 '19 at 18:03
  • $\begingroup$ Yes, that's the same. Usually, if the omnibus test is significant, then so is one of the pairwise differences. But these are different tests, so one need not imply the other (especially with borderline cases). The reason you'd do the omnibus test first is I guess as an extra safeguard against multiple testing. You only look for individual differences if you find a significant overall difference. I've also tried to explain the difference with ANOVA and separate pairwise testing here, if you're interested. $\endgroup$ Jul 6 '19 at 23:08
  • $\begingroup$ "Usually, if the omnibus test is significant, then so is one of the pairwise differences. But these are different tests, so one need not imply the other (especially with borderline cases). ... I've also tried to explain the difference with ANOVA and separate pairwise testing here, if you're interested." Thanks! $\endgroup$ Jul 7 '19 at 16:28


Yea, my previous answer is BS. Here is a Bayesian take as consolation.

There seems to bet some controversy about the multiple testing (and rightly so. I'm still researching). I think a quick and easy way to get around this is to do a bayesian logistic regression

effect_prior = prior('normal(0,0.5)', class = 'b')

model = brm(clicks|trials(N) ~ variant,
            data = experiment,
            family = binomial(),
            prior = c(effect_prior))

We know the effects can't be enormous. In online experiments, they are usually quite small. The prior reflects that. Results of the model are similar to the one above

Population-Level Effects: 
          Estimate Est.Error l-95% CI u-95% CI Eff.Sample Rhat
Intercept    -1.98      0.09    -2.16    -1.80       2396 1.00
variant_B     0.12      0.13    -0.12     0.38       2631 1.00
variant_C     0.37      0.12     0.13     0.61       2535 1.00

A credible interval for the difference between variant C and B is 0.010 to 0.475. Again, we are estimating that C is the best variant overall, and is likely to be better than B all things considered. Even if C was not better than B, C would still be the better option since we are quite certain B is not better than A

  • $\begingroup$ How does this answer the question? Using logistic regression you have exactly the same issues that with pairwise tests. $\endgroup$
    – Michael M
    Jul 6 '19 at 16:33
  • $\begingroup$ @MichaelM I'm presently looking up if a correction factor is required for confidence intervals derived in this manner. I'm inclined to believe that because I've already spent the degrees of freedom estimating the covariance matrix, I don't need the correction factor, but I could be wrong. Will edit my answer shortly if I find something to the contrary. $\endgroup$ Jul 6 '19 at 16:35
  • $\begingroup$ Thanks for the reply! Unfortunately, I'll need some time to familiarize myself with Bayesian approach to understand what is going on here :) But overall, this solution looks suspiciously simple. Are there any limitations? If there were 10 groups instead of 3, would it still work? How do I confidently select the best variant among them? $\endgroup$ Jul 7 '19 at 17:06
  • 1
    $\begingroup$ The Bayesian approach is deceptively simple. I found this paper which may elucidate some questions you may have. One caveat is that, in that paper, the authors make use of hierarchical models, which I do not use in this answer. However, a Bayesian approach isn't the de facto way to go, it is just something I offer as consolation. Even if you did all pairwise tests with no correction, your type one error would still be less than 10%. That isn't such a biog deal. It becomes a bigger deal with more groups. $\endgroup$ Jul 7 '19 at 17:14
  • 1
    $\begingroup$ @lost-in-space also, Frank Harell has a good explanation here as to why this can be done with Bayesian methods. $\endgroup$ Jul 7 '19 at 17:25

@Demetri Pananos, I agree that if you're just contrasting B and C then you do not need to correct. Because it's just one test. But you could just do a direct comparison of proportions, e.g.,

prop.test( c(150,180), c(1100,1070) ) 
data:  c(150, 180) out of c(1100, 1070)
X-squared = 4.0264, df = 1, p-value = 0.04479
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.0630087571 -0.0007125683
sample estimates:
   prop 1    prop 2 
0.1363636 0.1682243 

Which is a more powerful test.

If you wanted to contrast all three combinations then you technically should correct for multiple testing I guess, whether you use logistic regression or the binomial test given here. I don't think think I follow your logic that you've already estimated the covariance matrix therefore no multiple testing is required, because you're not doing significance testing for the entries of the covariance matrix. Perhaps you can explain more clearly.

  • $\begingroup$ "Which is a more powerful test. " -- Statistically speaking? If this is true it isn't much more powerful. $\endgroup$ Jul 6 '19 at 17:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.