In an intervention study, I have the following information - % of students passing an exam pre- and post intervention, sample size for control group and intervention group at pre- and post intervention, how can I work out the effect size?
1 Answer
$\begingroup$
$\endgroup$
11
This problem amounts to a $2\times 2$ contingency table (actually 2 such tables if you look at pre- and post-intervention). The variable for the columns could be the pass/fail counts, and the variable for the rows could be the treatment condition (control & intervention).
Assuming appropriate conditions have been met, this is a chi-square analysis, and the conventional effect size here is the phi-coefficient: $$\phi = \sqrt{\frac{\chi^2}{N}}$$ (Curiously, this also happens to be the correlation if you code the two variables as ones and zeros.)
The convention with this effect size is to classify with the cut-offs of 0.1, 0.3, and 0.5, for small, moderate, and large, respectively.
Update #1
Based on the information provided in the comments, this would better be described as a 4×2 contingency table. The first variable is the treatment condition (dichotomous), and the second variable is the the pre/post status description. You would have 4 conditions for this variable: achieved at pre and failed to achieve at post, failed to achieve at both pre and post, achieved at both pre and post, and failed to achieve at pre and achieved at post (ordered in "best" for substantiating treatment effect).
In this case, you would use the Cramer’s $V$ as the effect size measure: $$V = \sqrt{\frac{\chi^2}{N \cdot \text{min}(R-1,C-1)}}$$ where $R$ is the number of rows and $C$ is the number of columns in the table. This effect size is compared to the same cut-offs as the $phi$ coefficient.
-
1$\begingroup$ I'm sorry...I don't know what you mean by having a control group that has been exposed to the treatment...that's not a control group. $\endgroup$– Gregg HCommented May 5, 2023 at 14:44
-
1$\begingroup$ My first recommendation is NOT to use the counts, but to use the metric that is used to classify one as achieved or failed to achieve. My guess is that this is some scalar measurement. Then you can calculate the difference of these scores for each individual, and the matched-pairs effect size would be $d = \frac{\bar{d}}{s_d}$ where $\bar{d}$ is the average gain/loss score and $s_d$ is the standard deviation of these scores. However, if you must use counts, I have added an update to my original answer. $\endgroup$– Gregg HCommented May 8, 2023 at 19:17
-
1$\begingroup$ ¿Are there any sample sizes reported in the paper? If so, you can work the %s back to whole counts. $\endgroup$– Gregg HCommented May 12, 2023 at 15:49
-
1$\begingroup$ If you have the overall n and the percent breakdown for treatment and control, then if you have the percent and pre and post, you can indeed calculate all of the counts (within rounding). $\endgroup$– Gregg HCommented May 14, 2023 at 13:38
-
1$\begingroup$ We could also use Cohen's $\omega$ for this case (which is essentially $\phi$ for two-way tables larger than 2X2), but it depends a bit on what is the purpose of having an effect size in the first place. If it's for subsequent sample size calculations for future studies, Cohen's $\omega$ may be more appropriate, while on the other hand Cramér's V may be easier to interpret (if ease of interpretation is a feature we want). Anyway, one can easily convert Cramér's V to Cohen's $\omega$ (and conversely), so the choice is not such a big deal ultimately, but I think it's worth mentioning $\omega$. $\endgroup$– J-J-JCommented May 18, 2023 at 5:18