2
$\begingroup$

All sources I looked at say that correlation measures effect size. Hence, I presume, if $Y$ is a dependent variable of $X$ and $cor(X,Y)=1$ then the effect size of $X$ on $Y$ is considered huge? Perhaps even infinitely large? (since $1$ is the largest possible value of correlation). This doesn't make sense to me as the dependence between $X$ and $Y$ may be 100% certain but very small.

For example $X$ ranging between $0$ and $1$ and $Y=100+0.001X$ have correlation $1$, but the effect of $X$ on $Y$ is miniscule. (To make it more realistic, you can add a very small random error to $Y$, say of variance $10^{-10}.$)

$\endgroup$
3
  • $\begingroup$ If Y = 100 + 0.001X, then the change in X perfectly accounts for the change in Y, just as if the relationship were Y = 100 + 1,000,000X. Either way r = 1. "What makes sense" will vary from person to person and from time 1 to time 2. $\endgroup$
    – rolando2
    Apr 21, 2018 at 19:19
  • 3
    $\begingroup$ In view of the comments, I need to accept that statisticians label correlation as effect size. My concern is that it is not what effect size means to most people. I didn't do the polling but I imagine a newly discovered drug which extends human lifespan by 1 second with 100% certainty would not be deemed as having a huge effect on human lifespan, yet there is a perfect correlation between taking it and the lifespan. $\endgroup$
    – Adam
    Apr 21, 2018 at 22:13
  • $\begingroup$ Your effect size does not measure the extent of the effect on the human lifespan, hence your confusion. You can, in fact, have one that does: the 1 additional second is, in itself, an effect size. $\endgroup$
    – Firebug
    Sep 3, 2023 at 19:56

3 Answers 3

3
$\begingroup$

Adam, you raise a valid point. Effect size statistics are actually fairly heterogeneous with what information they convey. Probably the most prototypical is Cohen's d, which is what you might think an effect size statistic should be: it is a standardized measure of the difference between two means relative to the standard deviation of observations, as in a t-test. It doesn't, however, tell you anything about the absolute difference in means, which might be what is of actual importance.

Effect size statistics for nonparametric statistics like Wilcoxon-Mann-Whitney (r, Freeman's theta, epsilon-squared) are related to the probability that observations from one group are higher in rank than those from the other group. They can't tell you anything about the difference in, say, medians between the two groups, which you might desire an effect size statistic to convey, and which might be of actual importance.

All this weaves a confusing tapestry, for, say, students. These statistics called "effect size" each convey a different thing, although with a somewhat similar purpose. So, there are p-values that convey whether a test is "statistically significant", but there are limitations with how useful this is. Then there are effect size statistics, that are standardized in some way, and can be interpreted as "small", "medium", or "large", but these interpretations are very much relative to the field of study and the expectations of the study. Finally, there is what I call "practical importance". Often, this is the most subjective, but the most important practically. This is where you might consider the absolute difference in means, the absolute difference in medians, or the beta coefficient for a linear regression.

To some extent, this is just a problem of a statistic conveying only the information that that statistic conveys. (p-value vs. effect size vs. difference in means). Each tells you something, but none by itself is dispositive for reaching a statistical or practical conclusion.

$\endgroup$
1
  • 2
    $\begingroup$ Thanks Sal for your thoughtful comments. I just searched the web and, sadly, there is a huge number of books and papers equating effect size with practical significance. (That's what made me confused in the first place.) $\endgroup$
    – Adam
    Apr 22, 2018 at 18:58
1
$\begingroup$

Let's separate the magnitude of the correlation from one's level of certainty about it. If X = the two values {0,5} and Y = {10,15}, the observed correlation will be 1.00, indicating complete dependence or the maximum possible effect size that can appear using this metric. But who would feel certain about this based on just 2 observations?

Conversely, with ten million observations, a correlation of 0.05 would seem nearly certain to reflect a population correlation that's greater than zero, and its 95% or 99% confidence interval would be tiny. But .05 would indicate only a slight association between X and Y, and thus would constitute (in most contexts) a very small effect size.

$\endgroup$
2
  • $\begingroup$ I agree that 2 observations are not enough to establish certainty. But my question is not about that. (Assume that the "certainty" referred to in my question was established by millions of observations.) $\endgroup$
    – Adam
    Apr 21, 2018 at 18:09
  • $\begingroup$ I agree that correlation of $0.05$ is a small effect size. I have no problem with that. My confusion is about the situation when the correlation is $1$ or close to it. $\endgroup$
    – Adam
    Apr 21, 2018 at 18:56
0
$\begingroup$

There are two common effect sizes used in the ANOVA/multiple regression paradigms: Cohen's $d$ and $f$, and coefficients of determination $\eta^2$ or $R^2$. The former set deal with scale-adjusted differences in the means; the latter set deal with percent of variance accounted for by the independent variables.

The correlation you reference in your question is between the independent and dependent variable. However, you could also reference the correlation between the dependent variable and the predicted values for this variable using the independent variable(s), $R= \text{cor}(y,\hat{y})$. Note, $r^2 = R^2$. This value $R^2$ is the % of the variance of $y$ that is attributable to the independent variable(s). As this is "scaled" in the sense that you can only account for 0% to 100% of the total variation, this is a reasonable measure of effect size (ie, ¿what effect do the independent variables have on the dependent variable).

$\endgroup$
3
  • $\begingroup$ I understand your differentiation between $cor(Y,X)$ and $cor(Y,\hat Y)$. However I am confused by calling $cor(Y,\hat Y)$ a reasonable measure of effect size, since in my example it is 1 while the effect size seems miniscule. $\endgroup$
    – Adam
    Apr 21, 2018 at 18:52
  • $\begingroup$ If you are you are using a distance metric, then yes it is miniscule...and a choice like $d$ of $f$ would be more appropriate. However, if the measure is the accounted variance, then $R^2$ would be a reasonable effect size. (There are different ways to look at the "effects".) $\endgroup$
    – Gregg H
    Apr 21, 2018 at 19:57
  • 2
    $\begingroup$ Note that effect size can be standardized (e.g., r) or un-standardized (e.g., b = .001 or b = 1,000,000). In the case you present, whether b = .001 or 1,000,000, Y is perfectly predicted by X, so r = 1.00. If you standardized Y and X with Z scores, then regressing Zy on Zx will produce b_z = 1.00 in either situation in which b = .001 or b = 1,000,000. In short, r is a standardized way of reporting effect size, but un-standardized b may also be used as a measure of effect size. The choice depends on the metric of X and Y and whether r or b would make more sense to your audience. $\endgroup$
    – Bryan
    Apr 21, 2018 at 21:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.