1
$\begingroup$

I am concerned with the practical situation in which I design a 'no-budget' split-ballot 2 x 1 experiment. We have money for $n=80$ cases in total, thus $n=40$ in each of two groups. In total I will test a number of $J$ outcome variables for difference, either in proportions or in means. Thus I am concerned with the following set of hypotheses:

$$H_0: \mu_{jT}=\mu_{jC} \ \ \ \ \ \ \ \forall \ \ j=1,...,J$$

Which we may also express as

$$H_0: D_{1}=...=D_{J}=0$$

where $D_j=\mu_{jT}-\mu_{jC}$.

I know how to make a power analysis for means and proportions for a single test. For means I expect small or medium effect sizes. Using the package pwr in R I came to the following total n for independent sample t-test:

       .8 Power .9 Power
Small       787     1053
Medium      128      170
Large        51       68

Where for effect sizes the thresholds suggested by Cohen for his measure $d$ are given by .2, .5, and .8. I also estimated minimum required sample size for detecting a difference in proportion of .1, .2, and .3, with 80% power, which are $n=777$, $n=188$, and $n=79$, respectively (for this I used a conservative formula given in Agresti, 2002, Cateogrical data analysis).

So I conclude that I can only detect large effects $d$ with $n=80$ and proportion differences of .3 or higher with 80% power.

This is a nuisance given that I cannot increase sample size. So I wondered about the following: since I test $J$ variables simultanously and $H_0$ only needs to be rejected jointly (i.e., any $D$ unequal zero is evidence in favor of the alternative hypothesis), I feel that the power of my test is not that of a single test, but rather higher.

  • How should I evaluate $H_0$? It seems that $J$ independent tests are an option but a simultanous tests in a SEM modeling context or multivariate ANOVA is more appropriate.
  • How can I make a power calculation for $J$ independent tests?
  • How can I make a power calcualtion for a method testing $H_0$ simultanously?

Edit: I found this thread How to work out Effect Size for a MANOVA using G*Power. A power analysis for MANOVA can be done in GPower. But I have no intuition for a plausible choice of effect size measure f²(V) and the underlying Pillai's V. It also seems the correlation between the dependent variables would be important, but is not directly taken into account in Gpower.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.