# Coefficient of determination invariant to centering and rescaling of variables

Can someone provide a proof why the coefficient of determination given by

$$R^2:=1-\dfrac{||y-\hat{y}||_2^2}{||y-\overline{y}||_2^2}$$

in a multiple linear regression setting

$$y=X\beta+\epsilon$$

is invariant when standardizing (centering and dividing by the standard deviation) $X$ and $y$?

• Is this homework or self-study? May 29, 2018 at 11:56
• dear cherub its not related to a homework task.I often read that statement in the literature, but have never seen a proof for this, so I was interested in the proof. best May 29, 2018 at 12:29
• The result is not generally true: it holds only when a nonzero constant vector lies in the column space of $X$ (which is guaranteed when $X$ includes an "intercept" but otherwise might not hold).
– whuber
Nov 16, 2020 at 16:18

Denote $$\tilde{y}=\frac{y-\bar{y}}{\sigma_y}=\frac{Hy}{\sigma_y}$$ and $$\tilde{X}=HX\Sigma_x$$ the scaled variables (other than the scaled constant, which is evidently zero and drops out), where $$H=I-\iota\iota'/n$$ denotes the demeaning matrix and $$\Sigma_x$$ is a diagonal matrix with $$1/\sigma_{x_j}$$ on the main diagonal, $$j=1,\ldots,K$$.

Step 1: In the denominator of the $$R^2$$ of the regression of the scaled variables, we have $$||\tilde{y}-\overline{\tilde{y}}||_2^2=(\tilde{y}-\bar{\tilde{y}})'(\tilde{y}-\bar{\tilde{y}})=(y-\bar{y})'(y-\bar{y})/\sigma_y^2,$$ as demeaning the demeaned variable again will evidently do nothing.

Step 2: The numerator gives the sum of squared residuals of the regression. For the standard regression of $$y$$ on $$X$$, this well-known expression reads, in matrix notation, as $$y'(I-X(X'X)^{-1}X')y.\tag{1}$$

For the regression of the scaled variables, i.e., $$\tilde{y}$$ on $$\tilde{X}$$, we thus have $$||\tilde{y}-\hat{\tilde{y}}||_2^2=\frac{1}{\sigma_y^2}y'H[I-HX\Sigma_x(\Sigma_xX'HX\Sigma_x)^{-1}\Sigma_xX'H]Hy,$$ where it was used that $$H$$ is symmetric and idempotent and that $$\Sigma_x$$ is symmetric. Next, since $$\Sigma_x$$ is also invertible, the expression in square brackets simplifies to $$I-HX(X'HX)^{-1}X'H,$$ as $$(\Sigma_xX'HX\Sigma_x)^{-1}=\Sigma_x^{-1}(X'HX)^{-1}\Sigma_x^{-1}$$. (Incidentally, this argument reveals that scaling $$HX$$ by any invertible matrix $$A$$ rather than just $$\Sigma_x$$ would do the trick, as would dividing $$Hy$$ by any $$c\neq0$$.)

Thus, we obtain $$||\tilde{y}-\hat{\tilde{y}}||_2^2=\frac{1}{\sigma_y^2}y'H[I-HX(X'HX)^{-1}X'H]Hy,$$ Here, in analogy to (1), $$y'H[I-HX(X'HX)^{-1}X'H]Hy$$ denotes the sum of squared residuals of a regression of $$Hy$$ on $$HX$$. By the FWL theorem, these residuals are the same as those of a regression of $$y$$ on $$X$$ (provided $$X$$ contains a constant, or at least has columns that can be combined into a constant), i.e. $$||y-\hat{y}||_2^2=y'H[I-HX(X'HX)^{-1}X'H]Hy.$$

Hence, $$\begin{eqnarray*} R^2_{scaled}&:=&1-\dfrac{||\tilde{y}-\hat{\tilde{y}}||_2^2}{||\tilde{y}-\overline{\tilde{y}}||_2^2}\\ &=&1-\dfrac{||y-\hat{y}||_2^2/\sigma_y^2}{||y-\overline{y}||_2^2/\sigma_y^2}\\ &=&R^2 \end{eqnarray*}$$

For those who, like me, like numerical illustrations of formal results:

y <- runif(10)
X <- runif(10)

reg <- lm(y~X)

y.s <- scale(y)         # scaled y
X.s <- scale(X)         # scaled X
scale.reg <- lm(y.s~X.s)

y.d <- scale(y,scale=F) # demeaned y
X.d <- scale(X,scale=F) # demeaned X

> all.equal(summary(reg)$$r.squared, summary(scale.reg)$$r.squared)   # R^2s are the same
 TRUE

> all.equal(sum(resid(reg)^2)/sd(y)^2, sum(resid(scale.reg)^2))     # comparison numerators
 TRUE

> all.equal(sum((y-mean(y))^2)/sd(y)^2, sum((y.s-mean(y.s))^2))     # comparison denominators
 TRUE

> all.equal(resid(reg), resid(lm(y.d~X.d-1)), check.attributes = F) # residuals of regression and demeaned regression are the same
 TRUE

• Thank you very much Christoph Hank for your detailed proof ! May 29, 2018 at 13:27
• If you change the model to reg <- lm(y ~ X - 1) you will see there is a problem here.
– whuber
Nov 16, 2020 at 16:20
• Indeed, the argument works when $X$ contains a constant - which my notation, as I realize, may not make as clear as it should - but the appeal to FWL does mention the need to have a constant. Also, when computing $R^2$ it is generally recommended to have a constant (vector in the column space of $X$) as it would otherwise not even be guaranteed to be nonnegative. Nov 16, 2020 at 16:34