How to select the best variant from an A/B/C test?

Question

Suppose I've run an A/B/C experiment (same as A/B but with 3 groups instead of 2) and gathered the following data for number of participants in each group and number of desired actions in each group (e.g. clicks on a certain button):

\begin{array}{c|c|c|c} & a & b & c \\ \hline total & 1000 & 1100 & 1070 \\ clicks & 120 & 150 & 180 \end{array}

Conversion estimates are different for each group:

\begin{array}{c|c|c|c} conv & 0.120 & 0.136 & 0.168 \end{array}

How do I show the difference is statistically significant and select the best variant?

In A/B test with only two groups it is possible to compute distance between conversions and confidence interval using an equation

$$ conv_2 - conv_1 \pm t * \sqrt{ \frac{conv_1 (1 - conv_1)}{N_1} + \frac{conv_2 (1 - conv_2)}{N_2} } $$

where $t$-value is determined by desired confidence level ($t=1.96$ for $\alpha = 95 \%$ ). If the interval doesn't contain zero, then it is possible to select the version with the largest conversion, if the interval contains zero, then it is not possible to claim there is statistically significant difference between the two variants.

Is it still possible to perform pairwise comparison of A/B/C conversions using the equation above, but with $t$-value adjusted for multiple comparisons?

One possible adjustment is Bonferroni correction, where $t$-value for $(1 - (1 - \alpha) / m, \alpha = 0.95, m = 3 )$ confidence level should be used. This method is safe, but conservative.

Another method is Tukey's HSD where $t$-value should be replaced by $q$-value (e.g. from http://www.real-statistics.com/statistics-tables/studentized-range-q-table/ ). This is preferred over Bonferroni test.

So, what is a correct procedure to determine the best A/B/C-variant?

Answer 1

EDIT

Yea, my previous answer is BS. Here is a Bayesian take as consolation.

There seems to bet some controversy about the multiple testing (and rightly so. I'm still researching). I think a quick and easy way to get around this is to do a bayesian logistic regression

effect_prior = prior('normal(0,0.5)', class = 'b')

model = brm(clicks|trials(N) ~ variant,
            data = experiment,
            family = binomial(),
            prior = c(effect_prior))

We know the effects can't be enormous. In online experiments, they are usually quite small. The prior reflects that. Results of the model are similar to the one above

Population-Level Effects: 
          Estimate Est.Error l-95% CI u-95% CI Eff.Sample Rhat
Intercept    -1.98      0.09    -2.16    -1.80       2396 1.00
variant_B     0.12      0.13    -0.12     0.38       2631 1.00
variant_C     0.37      0.12     0.13     0.61       2535 1.00

A credible interval for the difference between variant C and B is 0.010 to 0.475. Again, we are estimating that C is the best variant overall, and is likely to be better than B all things considered. Even if C was not better than B, C would still be the better option since we are quite certain B is not better than A

Answer 2

@Demetri Pananos, I agree that if you're just contrasting B and C then you do not need to correct. Because it's just one test. But you could just do a direct comparison of proportions, e.g.,

prop.test( c(150,180), c(1100,1070) ) 
data:  c(150, 180) out of c(1100, 1070)
X-squared = 4.0264, df = 1, p-value = 0.04479
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.0630087571 -0.0007125683
sample estimates:
   prop 1    prop 2 
0.1363636 0.1682243 

Which is a more powerful test.

If you wanted to contrast all three combinations then you technically should correct for multiple testing I guess, whether you use logistic regression or the binomial test given here. I don't think think I follow your logic that you've already estimated the covariance matrix therefore no multiple testing is required, because you're not doing significance testing for the entries of the covariance matrix. Perhaps you can explain more clearly.