I want to compare the proportion of correct answers. My variables are categorical (group condition type and answer) and my answer options are dichotomic (one trial, one question - correct/ wrong). Previously I compared 2 groups and used chi-squared test for independent proportions, but now my design includes 3 different groups.

How can I determine the required sample size (a priori power analysis) in this situation, as in all power analysis calculators I can’t represent more than 2 groups (I tried G*Power, MATLAB, SPSS and R, and found only two-samples power analysis)?

Thank you

  • $\begingroup$ yes you can do the chi square test. don't know about sample size calculation though $\endgroup$
    – carlo
    Nov 8 '20 at 11:24
  • $\begingroup$ I do not understand this phrase "and my answer options are dichotomic (one trial, one question - correct/ wrong)". What do you mean by 'one trial, one question'? $\endgroup$ Nov 10 '20 at 17:39
  • $\begingroup$ Thanks. Each participant answers a question in a forced-choice screen. For each group only one of the answers in correct so the answers are classified in dichotomic way as “correct” or “wrong”. In “one trial, one question” I meant that participants see photos and answer the question only once (one round per participant), thus the “correctness” of their answers can’t be averaged. $\endgroup$
    – Nataly
    Nov 13 '20 at 7:35

Assuming the outcome is the proportion of correct individual answers to each of three independant questions or groups of questions without any ordering, one way to compute the sample size required to achieve a given statistical power is to rely on the chi-squared test. From there on, the effect size can be defined in two different ways.

The first approach is to consider the variance of the proportions divided by the product of the average proportion times one minus the average. For three proportions, this amounts to

$$ \delta = \sum_{i=1}^3 (p_i - \bar p)^2 \big/ (3\bar p(1- \bar p), $$

where $ \bar p = \big(\sum_{i=1}^3 p_i\big) / k$ (Ryan, T.P., Sample Size Determination and Power, Wiley, 2013). This is the approach used by nQuery Advisor.

Cohen (Statistical Power Analysis for the behavioral Sciences, 2nd ed., Lawrence Erlbaum Associates, 1988) discusses a different approach which relies on the chi-square statistic,

$$ \chi^2_{\text{obs}} = \sum_{i=1}^m \frac{(O_{ik} - E_{ik})^2}{E_{ik}}, $$

where $m$ denotes the number of cells, $i$ the row and $j$ the column of the table. We usually assume theoretical proportions $E_{ik}$ = 0.5, and the effect size is defined as the square root of the test statistic, $\chi^2_{\text{obs}}$ (section 7.2). The powerAnalysis R package implements this approach.

You can check that you get the same results as those discussed by Ryan, for the case of three theoretical proportions {0.4, 0.5, 0.6}, a statistical power of 0.90, and a $\chi^2_{\text{obs}}$ statistic with 2 degrees of freedom equal to 0.08 (p. 127):

power.chisq(sqrt(0.08), 2, power = 0.90, sig.level = 0.05)
## n = 159

Note that Cohen also provides sample size tables (p. 252 ff.).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.