Coefficient of determination invariant to centering and rescaling of variables

Question

Can someone provide a proof why the coefficient of determination given by

$$R^2:=1-\dfrac{||y-\hat{y}||_2^2}{||y-\overline{y}||_2^2}$$

in a multiple linear regression setting

$$y=X\beta+\epsilon$$

is invariant when standardizing (centering and dividing by the standard deviation) $X$ and $y$?

Answer 1

Denote $$\tilde{y}=\frac{y-\bar{y}}{\sigma_y}=\frac{Hy}{\sigma_y}$$ and $$\tilde{X}=HX\Sigma_x$$ the scaled variables (other than the scaled constant, which is evidently zero and drops out), where $H=I-\iota\iota'/n$ denotes the demeaning matrix and $\Sigma_x$ is a diagonal matrix with $1/\sigma_{x_j}$ on the main diagonal, $j=1,\ldots,K$.

Step 1: In the denominator of the $R^2$ of the regression of the scaled variables, we have $$||\tilde{y}-\overline{\tilde{y}}||_2^2=(\tilde{y}-\bar{\tilde{y}})'(\tilde{y}-\bar{\tilde{y}})=(y-\bar{y})'(y-\bar{y})/\sigma_y^2,$$ as demeaning the demeaned variable again will evidently do nothing.

Step 2: The numerator gives the sum of squared residuals of the regression. For the standard regression of $y$ on $X$, this well-known expression reads, in matrix notation, as $$y'(I-X(X'X)^{-1}X')y.\tag{1}$$

For the regression of the scaled variables, i.e., $\tilde{y}$ on $\tilde{X}$, we thus have $$ ||\tilde{y}-\hat{\tilde{y}}||_2^2=\frac{1}{\sigma_y^2}y'H[I-HX\Sigma_x(\Sigma_xX'HX\Sigma_x)^{-1}\Sigma_xX'H]Hy, $$ where it was used that $H$ is symmetric and idempotent and that $\Sigma_x$ is symmetric. Next, since $\Sigma_x$ is also invertible, the expression in square brackets simplifies to $$ I-HX(X'HX)^{-1}X'H, $$ as $(\Sigma_xX'HX\Sigma_x)^{-1}=\Sigma_x^{-1}(X'HX)^{-1}\Sigma_x^{-1}$. (Incidentally, this argument reveals that scaling $HX$ by any invertible matrix $A$ rather than just $\Sigma_x$ would do the trick, as would dividing $Hy$ by any $c\neq0$.)

Thus, we obtain $$ ||\tilde{y}-\hat{\tilde{y}}||_2^2=\frac{1}{\sigma_y^2}y'H[I-HX(X'HX)^{-1}X'H]Hy, $$ Here, in analogy to (1), $$ y'H[I-HX(X'HX)^{-1}X'H]Hy $$ denotes the sum of squared residuals of a regression of $Hy$ on $HX$. By the FWL theorem, these residuals are the same as those of a regression of $y$ on $X$ (provided $X$ contains a constant, or at least has columns that can be combined into a constant), i.e. $$ ||y-\hat{y}||_2^2=y'H[I-HX(X'HX)^{-1}X'H]Hy. $$

Hence, \begin{eqnarray*} R^2_{scaled}&:=&1-\dfrac{||\tilde{y}-\hat{\tilde{y}}||_2^2}{||\tilde{y}-\overline{\tilde{y}}||_2^2}\\ &=&1-\dfrac{||y-\hat{y}||_2^2/\sigma_y^2}{||y-\overline{y}||_2^2/\sigma_y^2}\\ &=&R^2 \end{eqnarray*}

For those who, like me, like numerical illustrations of formal results:

y <- runif(10)
X <- runif(10)

reg <- lm(y~X)

y.s <- scale(y)         # scaled y
X.s <- scale(X)         # scaled X
scale.reg <- lm(y.s~X.s)

y.d <- scale(y,scale=F) # demeaned y
X.d <- scale(X,scale=F) # demeaned X

> all.equal(summary(reg)$r.squared, summary(scale.reg)$r.squared)   # R^2s are the same
[1] TRUE

> all.equal(sum(resid(reg)^2)/sd(y)^2, sum(resid(scale.reg)^2))     # comparison numerators
[1] TRUE

> all.equal(sum((y-mean(y))^2)/sd(y)^2, sum((y.s-mean(y.s))^2))     # comparison denominators
[1] TRUE

> all.equal(resid(reg), resid(lm(y.d~X.d-1)), check.attributes = F) # residuals of regression and demeaned regression are the same
[1] TRUE