I am concerned with the practical situation in which I design a 'no-budget' split-ballot 2 x 1 experiment. We have money for $n=80$ cases in total, thus $n=40$ in each of two groups. In total I will test a number of $J$ outcome variables for difference, either in proportions or in means. Thus I am concerned with the following set of hypotheses:
$$H_0: \mu_{jT}=\mu_{jC} \ \ \ \ \ \ \ \forall \ \ j=1,...,J$$
Which we may also express as
$$H_0: D_{1}=...=D_{J}=0$$
where $D_j=\mu_{jT}-\mu_{jC}$.
I know how to make a power analysis for means and proportions for a single test. For means I expect small or medium effect sizes. Using the package pwr in R I came to the following total n for independent sample t-test:
.8 Power .9 Power
Small 787 1053
Medium 128 170
Large 51 68
Where for effect sizes the thresholds suggested by Cohen for his measure $d$ are given by .2, .5, and .8. I also estimated minimum required sample size for detecting a difference in proportion of .1, .2, and .3, with 80% power, which are $n=777$, $n=188$, and $n=79$, respectively (for this I used a conservative formula given in Agresti, 2002, Cateogrical data analysis).
So I conclude that I can only detect large effects $d$ with $n=80$ and proportion differences of .3 or higher with 80% power.
This is a nuisance given that I cannot increase sample size. So I wondered about the following: since I test $J$ variables simultanously and $H_0$ only needs to be rejected jointly (i.e., any $D$ unequal zero is evidence in favor of the alternative hypothesis), I feel that the power of my test is not that of a single test, but rather higher.
- How should I evaluate $H_0$? It seems that $J$ independent tests are an option but a simultanous tests in a SEM modeling context or multivariate ANOVA is more appropriate.
- How can I make a power calculation for $J$ independent tests?
- How can I make a power calcualtion for a method testing $H_0$ simultanously?
Edit: I found this thread How to work out Effect Size for a MANOVA using G*Power. A power analysis for MANOVA can be done in GPower. But I have no intuition for a plausible choice of effect size measure f²(V) and the underlying Pillai's V. It also seems the correlation between the dependent variables would be important, but is not directly taken into account in Gpower.
