Why do we use $R^2$ instead of $R$ in linear regression?

Question

$R^2$ equals the "amount of variance explained by the model".

However, we rarely use variance in descriptive statistics. We say a sample's weight is 78 ± 13 kg, which is $\bar x$ ± $\sigma$ (stdev), not $\bar x$ ± $\sigma^2$ (variance). This is because the standard deviation is in the same scale as the variable and is easier to understand than variance.

Thus my question: why don't we use "amount of standard deviation explained by the model"? Wouldn't you get it with $\sqrt{R^2}$? In my opinion, this would be much more intuitive to understand.

Answer 1

$R^2$ has several equivalent definitions. One is the squared correlation between the $x$ and $y$ variables in a simple linear regression. One is the squared correlation between the true values of $y$ and the predicted values; this has an advantage of working for multiple linear regression.

The definition that seems best-motivated to me involves the decomposition of the total sum of squares (SST) into the regression sum of squares (SSReg) and the residual sum of squares (SSRes). This $1-\frac{SSRes}{SST}$ expression can be considered on its own without thinking about the above two squared correlations. I think of that as a comparison of the performance of the fitted model to a naïve model that always predicts $\bar y$, no matter the feature values. Perhaps we could write $C=1-\frac{SSRes}{SST}$. However, it turns out that $C$ is equal to the above two squared correlations in OLS linear regression with an intercept, so it has been given the name $R^2$.

Answer 2

I have argued in many places, such as here, in favor of viewing $R^2$ as a comparison of the square loss incurred by your model to the square loss of a model that predicts the pooled mean $\bar y$ every time. To condense the notation, I want to give two definitions.

$$ SSRes = \overset{N}{\underset{i=1}{\sum}}\left( y_i-\hat y_i \right)^2\\ SSTotal = \overset{N}{\underset{i=1}{\sum}}\left( y_i-\bar y \right)^2 $$

Then we can view $R^2$ as follows, where the third of the fractions can be viewed as a ratio of the residual and total variance.

$$ R^2 = \left(\dfrac{ SSTotal - SSRes }{ SSTotal } \right)= 1-\left(\dfrac{ SSRes }{ SSTotal }\right) = 1-\left(\dfrac{ SSRes/N }{ SSTotal/N }\right) $$

If you take $+\sqrt{R^2}$, then you wind up with $\sqrt{1-\left(\dfrac{ SSRes/N }{ SSTotal/N }\right)}$.

If you want to view a statistic as comparing the standard deviations of the residuals and the total standard deviation, then the square root should be applied to the numerator and denominator separately.

$$ 1-\left(\dfrac{ \sqrt{SSRes/N} }{ \sqrt{SSTotal/N} }\right)=1-\sqrt{\dfrac{ SSRes/N }{ SSTotal/N }} $$

I cannot think of when I have seen this done, but it does not seem outrageous, as long as you are clear about your non-standard calculation and why you want to know this particular value. However: $$1-\sqrt{\dfrac{ SSRes/N }{ SSTotal/N }} \ne \sqrt{1-\left(\dfrac{ SSRes/N }{ SSTotal/N }\right)}$$

A simulation shows this.

set.seed(2023)
N <- 3
x <- seq(N)
y <- x + rnorm(N)
L <- lm(y ~ x)
ssres   <- sum((y - predict(L))^2)
sstotal <- sum((y - mean(y))^2)
sqrt(1 - (ssres/sstotal)) - (1 - sqrt(ssres/sstotal)) # I get 0.01926935, which I 
                                                      # find too high to be explained
                                                      # by floating-point errors
                                                      # 

Consequently, taking $+\sqrt{R^2}$ might not mean quite what you want it to mean.

Finally, the sign of $\sqrt{R^2}$ is ambiguous. It makes sense in a simple linear regression when there is just one feature $x$ and $\operatorname{cor}(x,y)$ makes perfect sense. However, if you have multiple features and regression coefficients with both positive and negative signs, the sign of $\sqrt{R^2}$ does not make sense to me.

If you want to consider the correlation between the predicted and observed values of $y$, there are issues with that, as it can miss discrepancies between the predicted and observed values that $R^2$, as it is defined above, does not miss. However, if you do find yourself in a situation where the correlation between predicted and true values is less than zero (it can happen in some of the more exotic modeling techniques than OLS linear regression with an intercept), you know something is not right about your predictions.