Open Science Collaboration (2015) is one of the most well-known and most-cited psychology papers of the past decade. In it the authors use an effect size I have never seen before, which they call “correlation coefficient per df”. It is used to facilitate the comparison of effect sizes across studies that used different test statistics.
The authors write:
We transformed effect sizes into correlation coefficients whenever possible. Correlation coefficients have several advantages over other effect size measures, such as Cohen’s d. Correlation coefficients are bounded, well known, and therefore more readily interpretable. Most important for our purposes, analysis of correlation coefficients is straightforward because, after applying the Fisher transformation, their standard error is only a function of sample size. Formulas and code for converting test statistics $z$, $F$, $t$, and $\chi^2$ into correlation coefficients are provided in the appendices at http://osf.io/ezum7.
In the relevant Appendix A3 (viewable at https://osf.io/z7aux/), Open Science Collaboration (2015) call their measure of effect size as “correlation coefficient per df”. They state on p.7:
Whenever possible, we calculated the “correlation coefficient per df” as effect size measure based on the reported test statistics. This was possible for the $z$, $\chi^2$ , $t$, and $F$ statistic.
They give the the following formula for the transformation of the $F$-statistic:
$r = \sqrt{\frac{F_{obs}\times\frac{df_1}{df_2}}{F_{obs}\times\frac{df_1}{df_2}+1}} \times \sqrt{\frac{1}{df_1}} $
, where $F_{obs}$ is the F-statistic observed in a particular study, and df1 and df2 are the numerator and denominator degrees of freedom respectively.
The authors claim the same transformation can be applied for t-statistics "where $F=t^2$", which I make out to be
$r = \sqrt{\frac{t^2_{obs} \times\frac{1}{df} } {\frac{t^2_{obs}}{df}+1}} $
The authors explain on p.7 of Appendix 3 that:
The expression in the first square-root equals the proportion of variance explained by the df1 predictors of the variance not yet explained by these same predictors. To take into account that more predictors can explain more variance, we divided this number by df1 to obtain the “explained variance by predictor”. Taking the square root gives the correlation, or more precisely, it gives the correlation of each predictor assuming that all df1 predictors contribute equally to the explained variance of the dependent variable.
Open Science Collaboration (2015) also state that
The z-statistic is transformed into a correlation using sample size N with $z = r_f \sqrt{(N-3)}$, with $r_f$ the Fisher-transformed correlation. The $\chi^2$ is transformed into the or correlation coefficient with $\phi = \sqrt{\frac{\chi^2}{N}}$.
I am also doing research in which I want to aggregate effect sizes across studies. For that, I’d also want to take different test statistics ($F$, $t$, $r$, $\chi^2$, etc) and their associated degrees of freedom and turn them into a common measure of effect size that facilitates aggregation over different studies. The “correlation coefficient per df” measure seems promising for my purposes, but I am a little spooked by the lack of information about it.
What is the logic behind this effect size measure? What does "the proportion of variance explained by the df1 predictors of the variance not yet explained by these same predictors" mean?
Is this measure's usage sound for comparing effect sizes across studies, where those effect sizes have been derived from ($z$, $\chi^2$, $F$, $t$) test statistics?
What are the origins of this effect size measure? It’s not apparent from Open Science Collaboration (2015) whether the authors adopted it from somewhere else or made it up. However, googling stuff like “correlation coefficient per df” doesn’t turn up any hits besides those that relate to that same 2015 paper.
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251).
