I need to do a hypothesis test with three versions (one control version and two test versions). From what I understand I can just run a t-test (or z-test) for each test version vs control, and then use some kind of correction (such a bonferroni correction) and see if one is significant (and if they both are see which one has the highest lift).

Assuming this is right, how do I run a sample size calculation for this? Can I just run the sample size calculation for both versions of the test (control vs test a and control vs test b) and then aggregate that way. I guess I would take the bonferroni correction into account when determining significance level for the sample size calculation. Does this seem right?


1 Answer 1


Maybe it is best to do an ANOVA to see if there are any significant differences among the three levels of the factor. If so, then do ad hoc tests along with some method (such as Bonferroni) to avoid 'false discovery'.

Unless I had some previous experience with similar data to feel confident that the three levels all have the same variability, I'd use oneway.test in R, which does not assume equal variances.


ctrl = rnorm(20, 50, 2)
trta = rnorm(20, 51, 2)
trtb = rnorm(20, 52, 2.2)
x = c(ctrl, trta, trtb)
g = rep(1:3, each=20)
boxplot(x~g, col="skyblue2")

enter image description here


    One-way analysis of means 
    (not assuming equal variances)

data:  x and g
F = 4.1362, num df = 2.000, denom df = 37.975, p-value = 0.02371

So there are significant differences at the 5% level. It seems obvious that the control and treatment B must differ. Then we could do ad hoc Welch t tests at the 2% level to see if we can distinguish between treatments A and B, and between control and treatment A. In my fictitious example, we cannot say treatment a is significantly different from the other two levels of the factor.

t.test(ctrl, trta)$p.val
[1] 0.06865211           # not signif
t.test(trta, trtb)$p.val
[1] 0.3030164            # not signif
t.test(ctrl, trtb)$p.val
[1] 0.007009848          # signif as surmised

There are procedures in R, other statistical software, and online to suggest sample sizes for the three levels, given the size of difference $\Delta$ we want to detect, the (common) SD $\sigma$ of the three groups, and the desired power. Most of these require equal variability in the levels and equal sample sizes. However, it is possible to relatively simple simulations in R to check whether particular sample sizes give adequate power for given $\Delta$ and $\sigma_i.$

In the fictitious situation above we might use $\sigma_1 = \sigma_2 = 2, \sigma_3 = 2.2,$ and $\Delta = 1.$ We found that $n = 20$ was large enough for the main ANOVA, but not large enough for the ad hoc tests (where smaller values of $\Delta$ may be required). So, as you suggest, we should focus on the power of the t tests.

By trial and error, it is easy to see that $n=80$ suffices to get power about 74% for a Welch t test with $\Delta=1, \sigma_1=2, \sigma_2-2.2,$ and significance level $\alpha = 2\%.$ (And $n-100$ for power about 84%.)

pv = replicate(10^5, t.test(rnorm(80,50,2),rnorm(80,51,2.2))$p.val)
mean(pv <= .02)
[1]  0.74339

pv = replicate(10^5, t.test(rnorm(100,50,2),rnorm(100,51,2.2))$p.val)
mean(pv <= .02)
[1] 0.84551

The search can be hurried along by (a) using $10^4$ replications early on and (b) using standard formulas involving noncentral t distributions for pooled t tests which have slightly better power than do Welch t tests.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.