The common language effect size (CLES) of McGraw & Wong, 1992, also known as the probability of superiority, can be conceptualized as the probability that a person picked at random from the treatment group will have a higher score than a person picked at random from the control group. Ruscio 2008 states that the conversion from Cohen's d to the CLES is $$ {\Phi}(\frac{d}{\sqrt{2}}) $$ where ${\Phi}$ is the cumulative distribution function of the standard normal distribution. As an example, they note that a Cohen's d of .8 is equivalent to a CLES of .71; the implementation of the formula on this online converter (which uses the terminology 'probability of superiority') seems to agree.

However, the online calculator in block 14 of this webpage uses a different calculation: specifically, it first converts d to r using the standard conversion formula $ r = \sqrt{\frac{d^2}{d^2 + 4}} $, and then r to CLES using the formula provided in Dunlap 1994, $$\frac{\arcsin{r}}{\pi} + 0.5$$

Converting a d of 0.8 to CLES by this method yields a very different result: 0.62.

I believe both approaches assume equal sample sizes. I am having trouble understanding why the methods yield different results. Is it that the two methods make different underlying assumptions? Are there arguments for using one of these conversions over another, if one's aim is to convert a population or sample d to a CLES which can be meaningfully interpreted as the probability that a person picked at random from the treatment group will have a higher score than a person picked at random from the control group?

Edit: I wonder if this note in the introduction of the Ruscio 2008 paper linked in the question provides a clue:

Although formulas exist to convert between d and rpb, McGrath and Meyer (2006) demonstrated that these measures sometimes prompt different conclusions. In particular, McGrath and Meyer showed that whereas the value of d is unaffected by two groups’ relative sample sizes, or base rates, 1 the value of rpb attains a maximum value with equal-sized groups and declines as sample sizes diverge.

The Dunlap 1994 paper refers to Pearson's r, as does the r in the standard formula $ r = \sqrt{\frac{d^2}{d^2 + 4}} $, not the point-biserial correlation. But perhaps a similar issue exists with Pearson's r, such that $ r = \sqrt{\frac{d^2}{d^2 + 4}} $ is merely an approximation? Any guidance appreciated.


Wrote the authors of both calculators, who both agreed that, if d needs to be converted to CLES in a context where one cannot calculate the CLES directly from the data, the Ruscio 2008 paper linked in my original question (and implemented here) is the preferred approach, as block 14 of the Psychometrica webpage uses an approximation. Furthermore, block 1 of the Psychometrica webpage has a calculator that computes the CLES from means and standard deviations.

If just wanting to compute the CLES from data from within R, canprot's CLES function probably the best way to go.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.