I am concerned with the practical situation in which I design a 'no-budget' split-ballot 2 x 1 experiment. We have money for $n=80$ cases in total, thus $n=40$ in each of two groups. In total I will test a number of $J$ outcome variables for difference, either in proportions or in means. Thus I am concerned with the following set of hypotheses:

$$H_0: \mu_{jT}=\mu_{jC} \ \ \ \ \ \ \ \forall \ \ j=1,...,J$$

Which we may also express as

$$H_0: D_{1}=...=D_{J}=0$$

where $D_j=\mu_{jT}-\mu_{jC}$.

I know how to make a power analysis for means and proportions for a single test. For means I expect small or medium effect sizes. Using the package pwr in R I came to the following total n for independent sample t-test:

       .8 Power .9 Power
Small       787     1053
Medium      128      170
Large        51       68

Where for effect sizes the thresholds suggested by Cohen for his measure $d$ are given by .2, .5, and .8. I also estimated minimum required sample size for detecting a difference in proportion of .1, .2, and .3, with 80% power, which are $n=777$, $n=188$, and $n=79$, respectively (for this I used a conservative formula given in Agresti, 2002, Cateogrical data analysis).

So I conclude that I can only detect large effects $d$ with $n=80$ and proportion differences of .3 or higher with 80% power.

This is a nuisance given that I cannot increase sample size. So I wondered about the following: since I test $J$ variables simultanously and $H_0$ only needs to be rejected jointly (i.e., any $D$ unequal zero is evidence in favor of the alternative hypothesis), I feel that the power of my test is not that of a single test, but rather higher.

  • How should I evaluate $H_0$? It seems that $J$ independent tests are an option but a simultanous tests in a SEM modeling context or multivariate ANOVA is more appropriate.
  • How can I make a power calculation for $J$ independent tests?
  • How can I make a power calcualtion for a method testing $H_0$ simultanously?

Edit: I found this thread How to work out Effect Size for a MANOVA using G*Power. A power analysis for MANOVA can be done in GPower. But I have no intuition for a plausible choice of effect size measure f²(V) and the underlying Pillai's V. It also seems the correlation between the dependent variables would be important, but is not directly taken into account in Gpower.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.