2
$\begingroup$

I just want to make sure I have something clear in my head. When I calculate the effect size for a paired samples t-test after obtaining a significant result, I simply take the mean of the differences divided by the standard deviation of the differences to get an effect size d. Do I then need to take d and divide it by the square root of 1-r, where r is the correlation between pairs and r is estimated from the sample pairs? I am confused because dividing by the square root of 1-r supposedly gives me an "operative effect size" and I'm not really sure if the operative effect size is what I should be reporting in my analyses. For example, in this report I am working on, I need to know if there was an effect size of 2 SD. So when I calculate my effect size, should I be dividing by square root 1-r? I don't think so, I think I need to report the actual detected effect size and not the operative effect, but I would love a second opinion. Thanks!

$\endgroup$
  • $\begingroup$ I've not heard of this operative effect size. There are equations using 1-r but I'm uncertain if you have the right one. Do you have references for operative effect size and your 1-r including equation? $\endgroup$ – John Oct 26 '13 at 14:44
  • $\begingroup$ Yeah sure, it was brought up and discussed in an earlier question I had. stats.stackexchange.com/questions/71525/… $\endgroup$ – Jimj Oct 26 '13 at 15:36
  • $\begingroup$ @John As far as I know, Jacob Cohen coined the term in his popular power analysis textbook (1971, 1988). $\endgroup$ – Jake Westfall Oct 26 '13 at 17:50
2
$\begingroup$

If you are constructing $d$ as the mean difference divided by the standard deviation of the difference scores -- rather than by the pooled standard deviation of scores in each group, as $d$ is conventionally defined! -- then that is already what I referred to (following Cohen, 1988) as the "operative effect size." So further dividing this operative effect size by $\sqrt{1-r}$ would not make sense, because that correction is already "built in" to that instantiation of $d$.

I think @John has a good brief discussion of different ways of computing $d$ at the bottom of his answer HERE. John mentions that some people firmly believe that $d$ should always be computed using the classical, independent-groups specificaton. I am one of these people. (Cohen was also one of these people. That's why he used the separate term "operative effect size" to talk about other ways of computing $d$.) I think it is a very bad idea to give $d$ different definitions in different contexts. Aside from the important problem this creates of killing any possible comparison of $d$ sizes between experimental paradigms that tend to use different designs, this inconsistent definition of $d$ also fosters confusion about what any given person means when they speak of $d$, unless they explicitly say which $d$ they mean! I believe this latter confusion is exactly what we have experienced here.

The "operative effect size" language convention is an attempt to allow us to talk sensibly and unambiguously about these nonstandard, but still useful, definitions of $d$ (nonstandard in the sense that they deviate from Cohen's definition). In case you are wondering, I believe Cohen calls these "operative" effect sizes because they are the effect sizes that are relevant for conducting a power analysis, which is what makes them useful. But let's keep in mind that this is only one of the uses of an effect size measure.

  • Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd edition). Routledge.
$\endgroup$
  • $\begingroup$ Hi Jake, thanks for that. So just to be clear, if I calculate an effect size d as the mean difference divided by the standard deviation of the difference scores, is that the same as calculating d in the conventional way and the dividing by square root 1-r? $\endgroup$ – Jimj Oct 26 '13 at 20:52
  • 1
    $\begingroup$ @Jimj Yes, that's right. $\endgroup$ – Jake Westfall Oct 26 '13 at 21:29
  • $\begingroup$ HI Jake, I just had one more question for you. Are you sure dividing the mean difference by the SD of the difference isn't the same as taking the classical d and dividing by sqrt(1-r^2)? I tried comparing the two methods on the following dataset: group a= 6,6,8,8,9 and group b= 1,2,3,4,4. The r squared version seemed to match up better. $\endgroup$ – Jimj Oct 30 '13 at 8:04
  • $\begingroup$ @Jimj I am sure the denominator contains $\sqrt{1-r}$ and not $\sqrt{1-r^2}$. The formula is copied from Cohen (1988), and can also be found (with discussion) on p. 264 of this text ( goo.gl/0SUqMr ), and I have verified it using some pen-and-paper computations. And I believe Glen_b also verified it in the comments of one of your previous questions. So with all that said, I admit I am at a loss as to why the two computational approaches are not giving the same numerical result in your simple example. We are probably overlooking something obvious. But I am not sure what at the moment... $\endgroup$ – Jake Westfall Oct 30 '13 at 21:59
  • 1
    $\begingroup$ @Jimj I believe I figured this out. The s.d. of the differences is not $\sigma\sqrt{1-r}$ but rather $\sigma\sqrt{2(1-r)}$. This makes the two computations agree on your small example (except for a small amount of bias). The reason the denominator of the operative effect size uses $\sigma\sqrt{1-r}$ and not $\sigma\sqrt{2(1-r)}$ is a little complicated to explain in a short comment, but the basic issue is that the power formulae derived for the independent-groups case only work nicely in the paired case if you factor out the $1/\sqrt{2}$, because that already appears elsewhere in the formulae. $\endgroup$ – Jake Westfall Nov 1 '13 at 6:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.