6
$\begingroup$

I have performed a Cochran's Q test for a within-subjects experimental design with 3 conditions and 36 participants with a dichotomous dependent variable.

I found a (just) statistically significant effect ($\chi^2$ = 6.00, df = 2, p = 0.04979) and would like to also report the effect size, but haven't been able to find any information as to which effect measure to use and how to calculate it.

Any pointers would be gratefully received - the domain is human factors (psychology).

$\endgroup$
3
$\begingroup$

I have found two measures of effect size in the literature for Cochran's $Q$ test of $b$ blocks (subjects) and $k$ treatments (groups):

Serlin, Carr and Marascuillo's (2007) maximum-corrected measure of effect size ($\eta^{2}_{Q}$), which is given by:

$$\eta^{2}_{Q} = \frac{Q}{b(k-1)},$$

where $0\le\eta^{2}_{Q}\le 1$.

Berry, Johnston and Mielke (2007) offer a chance-corrected measure of effect size ($\mathcal{R}$), which is given by:

$$\mathcal{R} = 1 - \frac{\delta}{\mu_{\delta}},$$

where:

$$\delta = \left[k {b\choose 2}\right]^{-1}\sum_{i=1}^{k}\sum_{j=1}^{b-1}\sum_{l=j+1}^{b}{\left|x_{ji}-x_{li}\right|}$$

for observations $x$ in data matrix $\mathbf{X}$,

$$\mu_{\delta} = \frac{2}{b\left(b-1\right)}\left[\left(\sum_{i=1}^{b}{p_{i}}\right)\left(b-\sum_{i=1}^{b}{p_{i}}\right)-\sum_{i=1}^{n}{p_{i}\left(1-p_{i}\right)}\right],$$

and $p_{i}$ is the proportions of successes across all treatments in the $i^{\text{th}}$ block.

Update: From personal correspondence with Berry, the published Equation [7] contains a typographical error, and the $2/[k(k-1)]$ term in the equation for $\mu_{\delta}$ should be replaced with $2/[b(b-1)]$ as I have represented above.

Berry &Co. also make a critique of $\eta^{2}_{Q}$ versus $\mathcal{R}$, writing (I substitute the symbols $b$ and $k$ for the symbols $n$ and $c$ appearing in their paper):

Chance-corrected measures of effect size, such as $\mathcal{R}$, possess distinct advantages in interpretation over maximum-corrected measures of effect size, such as $\eta^{2}_{Q}$. The problem lies in the manner in which $\eta^{2}_{Q}$ is maximized. The denominator of $\eta^{2}_{Q}$, $Q_{\max}=b(k-1)$, standardizes the observed value of $Q$ for the sample size and the number of treatments. Unfortunately, $b(k-1)$ does not standardize $Q$ for the data on which $Q$ is based but rather standardizes $Q$ on another unobserved hypothetical set of data.

A little farther, they sell the merits of $\mathcal{R}$ over those of $\eta^{2}_{Q}$:

$\mathcal{R}$ is completely data dependent, whereas $\eta^{2}_{Q}$ relies on an unobserved, idealized data set for its maximum value. Thus, $\mathcal{R}$ can achieve an effect size of unity for the observed data, while this is usually impossible for $\eta^{2}_{Q}$. Second, $\mathcal{R}$ is a chance-corrected measure of effect size. Furthermore, $\mathcal{R}$ is zero under chance conditions, unity when agreement among the $b$ subjects is perfect, and negative under conditions of disagreement. Therefore, $\mathcal{R}$ has a clear interpretation corresponding to Cohen's coefficient of agreement (1960) and other chance-corrected measures that is familiar to most researchers. On the other hand, $\eta^{2}_{Q}$ possesses no meaningful interpretation except for values of 0 and 1. Although takes the form of a correlation ratio, it cannot be interpreted as a correlation coefficient unless the marginal frequency totals are identical

I have implemented both of these effect size measures in Stata in the cochranq package, which can be accessed within Stata by typing net describe cochranq, from(https://alexisdinno.com/stata). The nonpar package for R on CRAN contains the cochrans.q program, which will give the classical *Q test, but does not offer the more recent and precise non-asymptomatic tests statistic, or effect size calculations, or adjustments for multiple comparisons.


References
Berry, K. J., Johnston, J. E., and Paul W. Mielke, J. (2007). An alternative measure of effect size for Cochran’s $Q$ test for related proportions. Perceptual and Motor Skills, 104:1236–1242.

Serlin, R. C., Carr, J., and Marascuillo, L. A. (1982). A measure of association for selected nonparametric procedures. Psychological Bulletin, 92:786–790.

$\endgroup$
2
$\begingroup$

I found this paper with Google but I cannot access it, so I don't really know what it is about really:

Berry KJ, Johnston JE, Mielke PW Jr. An alternative measure of effect size for Cochran's Q test for related proportions. Percept Mot Skills. 2007 Jun;104(3 Pt 2):1236-42.

I initially thought that using pairwise multiple comparisons with Cochran or McNemar test* (if the overall test is significant) would give you further indication of where the differences lie, while reporting simple difference for your binary outcome would help asserting the magnitude of the observed difference.

* I found an online tutorial with R.

$\endgroup$
  • $\begingroup$ Yes, I saw that paper too, but couldn't access it :( In the end I did use McNemar's test to do pairwise tests, for which I reported effect sizes, however for the original Cochran's Q test, I never did find a suitable effect size measure. I think it is convention to report effect sizes, both for the original test and for the post-hoc pairwise tests, right? $\endgroup$ – Ham May 25 '11 at 8:36
  • $\begingroup$ @Ham Yes, reporting ESs is recommended in many settings (I can think of the APA guidelines for example), although I would say reporting group difference (unstandardized, because ES are just standardized difference) is already good practice. $\endgroup$ – chl May 25 '11 at 8:53

protected by kjetil b halvorsen Oct 12 '17 at 1:17

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.