3
$\begingroup$

I'm sure you know this sample size calculator by Evan Miller. Given necessary parameters, it outputs the efficient sizes for Control & Test groups.

But what if I have multiple Test groups? What should be their sample sizes?

Ideally, the solution should take multiple testing corrections into account, but it is not required.

I would love to see derivations!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

The sample size calculations are the same as the standard RCT case when using multiple groups. Therefore, if we require ~600 users per group to detect a particular change in proportion with a given statistical power and significance level, if we have 5 variation we will require ~3,000 users. What changes is exactly the point you raised about multiple correction.

Assuming we do not have a multitude of independent A/B test (e.g. <10) Bonferroni correction is actually pretty decent. For example, the probability of at least one significant result when we have 5 tests and we use Bonferroni correction to control for multiple comparison is: $1-P(\text{No significant result})$ $\Rightarrow $ $1 - (1 - \frac{0.05}{5})^5 $ $=$ $0.0490$; for practical terms this (new) family-wise error rate based on the adjusted level of $\frac{\alpha}{k}$ is usually fine. The power will suffer though because the associated $z_{\alpha/2}$ will immediately get much more extreme. It would be uncommon to go from 0.80 to 0.50 just by correcting for 5 tests. Chow et al. (2008) "Sample Size Calculations in Clinical Research" Ch. 4: Large Sample Tests for Proportions is a really good resource on the matter if you wish to explore this behaviour further, as well as the package pwr in R.

When we have more comparisons the Bonferroni correction is very likely too conservative (i.e. our Type II error rate gets too high). This leads to requiring a potentially unreasonably large sample size to detect a fixed difference with given statistical power. The main option is to usually use FDR methods - control for the False Discovery Rate (which is a huge subject). As an immediate middle ground we might want to use Holm-Bonferroni method, which is just as general as Bonferroni but less conservative. Again, we can explore this behaviour with R using the stats::p.adjust which is immediately available in every R installation.

Finally, notice that in the context of A/B test for a website we often look into multi-armed bandit (MAB) optimization. MAB is not focusing on statistical significance but rather on exploitation of traffic. It relates much closer to Thompson Sampling rather than an RCT; Russo et al. (2017) "A Tutorial on Thompson Sampling" is probably the most complete tutorial on matter.

Improve this answer
$\endgroup$
6
  • $\begingroup$ The family-wise error rate may not be entirely correct. There should be no division by 5 I expect. $\endgroup$
    – StatsBio
    Commented Dec 16, 2021 at 18:28
  • $\begingroup$ Thank you for looking into this. When using Bonferroni correction we are setting the level of the pairwise tests required to declare statistical significance equal to $\frac{\alpha}{k}$. I suspect you are thinking of Sidak's procedure where the level is $1-(1-\alpha)^{\frac{1}{k}}$ (e.g. See Regression Methods in Biostatistics by Vittinghoff et al. Sect. 4.3.4 "Multiple Pairwise Comparisons Between Categories" for more details.) $\endgroup$
    – usεr11852
    Commented Dec 16, 2021 at 20:54
  • $\begingroup$ To put it otherwise: the whole motivation for FWER is that through Bonferroni we are able to get something close to our notional idea of significant level of 0.05 after all our testing is complete. (Sidak's procedure brings us actually exactly at our desired experiment-wise Type-I Error Rate) $\endgroup$
    – usεr11852
    Commented Dec 16, 2021 at 21:25
  • $\begingroup$ I would think that perhaps the alpha for Bonferroni correction should be specified as at the level "per comparison" e,g,. The equation is shown in the following video and does not appear to be that of Sidak's procedure youtube.com/watch?v=rMuNniCTsOw $\endgroup$
    – StatsBio
    Commented Dec 17, 2021 at 12:27
  • $\begingroup$ Thank you for adding that video, I understand what you mean now. (Yep, not Sidak.) The video refers to the FWER (~at the 04:30 mark) if we don't use the Bonferroni correction to reduce our $\alpha_{PC}$ to account for multiple comparisons. I refer to "this family-wise error rate" , i.e. when using the Bonferroni correction to get our new $a^*_{PC}$. I will amend the wording slightly to make this more clear. Thank you taking the time to read this carefully. $\endgroup$
    – usεr11852
    Commented Dec 17, 2021 at 13:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.