# ABC test/ hypothesis testing with multiple comparisons

I need to do a hypothesis test with three versions (one control version and two test versions). From what I understand I can just run a t-test (or z-test) for each test version vs control, and then use some kind of correction (such a bonferroni correction) and see if one is significant (and if they both are see which one has the highest lift).

Assuming this is right, how do I run a sample size calculation for this? Can I just run the sample size calculation for both versions of the test (control vs test a and control vs test b) and then aggregate that way. I guess I would take the bonferroni correction into account when determining significance level for the sample size calculation. Does this seem right?

Maybe it is best to do an ANOVA to see if there are any significant differences among the three levels of the factor. If so, then do ad hoc tests along with some method (such as Bonferroni) to avoid 'false discovery'.

Unless I had some previous experience with similar data to feel confident that the three levels all have the same variability, I'd use oneway.test in R, which does not assume equal variances.

Example:

ctrl = rnorm(20, 50, 2)
trta = rnorm(20, 51, 2)
trtb = rnorm(20, 52, 2.2)
x = c(ctrl, trta, trtb)
g = rep(1:3, each=20)
boxplot(x~g, col="skyblue2") oneway.test(x~g)

One-way analysis of means
(not assuming equal variances)

data:  x and g
F = 4.1362, num df = 2.000, denom df = 37.975, p-value = 0.02371


So there are significant differences at the 5% level. It seems obvious that the control and treatment B must differ. Then we could do ad hoc Welch t tests at the 2% level to see if we can distinguish between treatments A and B, and between control and treatment A. In my fictitious example, we cannot say treatment a is significantly different from the other two levels of the factor.

t.test(ctrl, trta)$$p.val  0.06865211 # not signif t.test(trta, trtb)$$p.val
 0.3030164            # not signif
t.test(ctrl, trtb)$p.val  0.007009848 # signif as surmised  There are procedures in R, other statistical software, and online to suggest sample sizes for the three levels, given the size of difference $$\Delta$$ we want to detect, the (common) SD $$\sigma$$ of the three groups, and the desired power. Most of these require equal variability in the levels and equal sample sizes. However, it is possible to relatively simple simulations in R to check whether particular sample sizes give adequate power for given $$\Delta$$ and $$\sigma_i.$$ In the fictitious situation above we might use $$\sigma_1 = \sigma_2 = 2, \sigma_3 = 2.2,$$ and $$\Delta = 1.$$ We found that $$n = 20$$ was large enough for the main ANOVA, but not large enough for the ad hoc tests (where smaller values of $$\Delta$$ may be required). So, as you suggest, we should focus on the power of the t tests. By trial and error, it is easy to see that $$n=80$$ suffices to get power about 74% for a Welch t test with $$\Delta=1, \sigma_1=2, \sigma_2-2.2,$$ and significance level $$\alpha = 2\%.$$ (And $$n-100$$ for power about 84%.) pv = replicate(10^5, t.test(rnorm(80,50,2),rnorm(80,51,2.2))$p.val)
mean(pv <= .02)
  0.74339

pv = replicate(10^5, t.test(rnorm(100,50,2),rnorm(100,51,2.2))\$p.val)
mean(pv <= .02)
 0.84551


The search can be hurried along by (a) using $$10^4$$ replications early on and (b) using standard formulas involving noncentral t distributions for pooled t tests which have slightly better power than do Welch t tests.