What does $R^2$ tell and what is the “strength of the relationship” between two variables?

Question

I was reading this answer by @whuber where it is discussed about the meaning and uses of $R^2$.

I’m having some trouble understanding what the “strength of the relationship” between two variables truly means.

Let’s say we have $Y$=“boat speed” and $X$=“wind speed”. We could use this model: $Y=aX+ε$ to express the relationship. (I know this isn’t true but that’s not the point I want to make)

I would intuitively think that the strength of the relationship is about: (1) how much influence $X$ has on $Y$ (in other words, how much $Y$ changes when $X$ does) and (2) how strong this influence is.

So I would say that the strength increases if $Var(\epsilon)$ decreases (2), and that it increases if the slope $a$ does (1). This is obviously dependent on the unit of measure so it should be adjusted in some way.

Would you agree this is a correct interpretation?

I also intuitively understand that the strength shouldn’t be influenced by the range of $X$: the relationship between wind speed and boat speed isn’t different if we test it in a place where the wind is more or less variable.

Is this the reason why saying that “$R^2 = \frac{a^2 Var[x]}{a^2 Var[X] + Var[\epsilon]}$ quantifies the strength of the relationship” is a bit of a stretch? (Since it is greatly affected by $Var(X)$ and the strength shouldn’t)

Would you say that $R^2$ is a statistic dependent on (1)The strength of the relationship (which, again, I would say that is only influenced by $VAR(\epsilon)$ and $a$) and (2)$Var(X)$?

If so, we could say that $R^2$ represents the strength of the influence that the relationship has on $Y$?

(I could see why the strength of the influence depends both on the strength of the relationship and the range of the predictor: the strength of the influence that wind speed has on boat speed depends on how strong the relationship between the two is and on how much the wind varies)

Answer 1

If it makes things any easier, consider that $R^2 = cor(X, Y)^2$, where $cor(X, Y)$ is the Pearson correlation coefficient between $X$ and $Y$. In turn, $cor(X,Y)$ is the normalized covariance between $X$ and $Y$ -- it is normalized so as to be between -1 and 1:

$cor(X,Y) = \frac{Cov(X,Y)}{sd_X sd_Y}$

Notice that if $X_1$ and $X_2$ only differ for their variance, we can write $X_1 = b X_2$, where $b \neq \pm 1$ is a scalar. Then, let $Y_1 = aX_1 + \epsilon$ and $Y_2 = aX_2 + \epsilon = abX_1 + \epsilon$. You are asking if $cor(aX_1, Y_1)$ and $cor(abX_1, Y_2)$ are different.

We can use the standard properties of $Cov(X,Y)$, $Var(X)$ ($sd_X$) and the OLS assumption that $\epsilon$ and $X_1$ are independent to find out that:

$cor(aX_1,Y_1) = \frac{a Var(X_1)}{sd_{X_1} sd_{Y_1}}$

$cor(abX_1,Y_2) = \frac{ab Var(X_1)}{sd_{X_1} sd_{Y_2}}$

The two quantities are different indeed.

That said if you multiply both $X_1$ and $\epsilon$ by $b$, then $R^2$ will not change. Such multiplication of both terms is what is really happening in your boat example. If the wind goes up (down) both your wind speed and your error will go up (down), leaving the strength of the relationship unchanged. Think what happens when $Y = aX$ with no error. It doesn't matter what is the variance of $X$, your correlation will always be perfect.