6
$\begingroup$

Can someone provide a proof why the coefficient of determination given by

$$R^2:=1-\dfrac{||y-\hat{y}||_2^2}{||y-\overline{y}||_2^2}$$

in a multiple linear regression setting

$$y=X\beta+\epsilon$$

is invariant when standardizing (centering and dividing by the standard deviation) $X$ and $y$?

$\endgroup$
3
  • $\begingroup$ Is this homework or self-study? $\endgroup$
    – cherub
    Commented May 29, 2018 at 11:56
  • 3
    $\begingroup$ dear cherub its not related to a homework task.I often read that statement in the literature, but have never seen a proof for this, so I was interested in the proof. best $\endgroup$
    – franzfranz
    Commented May 29, 2018 at 12:29
  • $\begingroup$ The result is not generally true: it holds only when a nonzero constant vector lies in the column space of $X$ (which is guaranteed when $X$ includes an "intercept" but otherwise might not hold). $\endgroup$
    – whuber
    Commented Nov 16, 2020 at 16:18

1 Answer 1

4
$\begingroup$

Denote $$\tilde{y}=\frac{y-\bar{y}}{\sigma_y}=\frac{Hy}{\sigma_y}$$ and $$\tilde{X}=HX\Sigma_x$$ the scaled variables (other than the scaled constant, which is evidently zero and drops out), where $H=I-\iota\iota'/n$ denotes the demeaning matrix and $\Sigma_x$ is a diagonal matrix with $1/\sigma_{x_j}$ on the main diagonal, $j=1,\ldots,K$.

Step 1: In the denominator of the $R^2$ of the regression of the scaled variables, we have $$||\tilde{y}-\overline{\tilde{y}}||_2^2=(\tilde{y}-\bar{\tilde{y}})'(\tilde{y}-\bar{\tilde{y}})=(y-\bar{y})'(y-\bar{y})/\sigma_y^2,$$ as demeaning the demeaned variable again will evidently do nothing.

Step 2: The numerator gives the sum of squared residuals of the regression. For the standard regression of $y$ on $X$, this well-known expression reads, in matrix notation, as $$y'(I-X(X'X)^{-1}X')y.\tag{1}$$

For the regression of the scaled variables, i.e., $\tilde{y}$ on $\tilde{X}$, we thus have $$ ||\tilde{y}-\hat{\tilde{y}}||_2^2=\frac{1}{\sigma_y^2}y'H[I-HX\Sigma_x(\Sigma_xX'HX\Sigma_x)^{-1}\Sigma_xX'H]Hy, $$ where it was used that $H$ is symmetric and idempotent and that $\Sigma_x$ is symmetric. Next, since $\Sigma_x$ is also invertible, the expression in square brackets simplifies to $$ I-HX(X'HX)^{-1}X'H, $$ as $(\Sigma_xX'HX\Sigma_x)^{-1}=\Sigma_x^{-1}(X'HX)^{-1}\Sigma_x^{-1}$. (Incidentally, this argument reveals that scaling $HX$ by any invertible matrix $A$ rather than just $\Sigma_x$ would do the trick, as would dividing $Hy$ by any $c\neq0$.)

Thus, we obtain $$ ||\tilde{y}-\hat{\tilde{y}}||_2^2=\frac{1}{\sigma_y^2}y'H[I-HX(X'HX)^{-1}X'H]Hy, $$ Here, in analogy to (1), $$ y'H[I-HX(X'HX)^{-1}X'H]Hy $$ denotes the sum of squared residuals of a regression of $Hy$ on $HX$. By the FWL theorem, these residuals are the same as those of a regression of $y$ on $X$ (provided $X$ contains a constant, or at least has columns that can be combined into a constant), i.e. $$ ||y-\hat{y}||_2^2=y'H[I-HX(X'HX)^{-1}X'H]Hy. $$

Hence, \begin{eqnarray*} R^2_{scaled}&:=&1-\dfrac{||\tilde{y}-\hat{\tilde{y}}||_2^2}{||\tilde{y}-\overline{\tilde{y}}||_2^2}\\ &=&1-\dfrac{||y-\hat{y}||_2^2/\sigma_y^2}{||y-\overline{y}||_2^2/\sigma_y^2}\\ &=&R^2 \end{eqnarray*}

For those who, like me, like numerical illustrations of formal results:

y <- runif(10)
X <- runif(10)

reg <- lm(y~X)

y.s <- scale(y)         # scaled y
X.s <- scale(X)         # scaled X
scale.reg <- lm(y.s~X.s)

y.d <- scale(y,scale=F) # demeaned y
X.d <- scale(X,scale=F) # demeaned X

> all.equal(summary(reg)$r.squared, summary(scale.reg)$r.squared)   # R^2s are the same
[1] TRUE

> all.equal(sum(resid(reg)^2)/sd(y)^2, sum(resid(scale.reg)^2))     # comparison numerators
[1] TRUE

> all.equal(sum((y-mean(y))^2)/sd(y)^2, sum((y.s-mean(y.s))^2))     # comparison denominators
[1] TRUE

> all.equal(resid(reg), resid(lm(y.d~X.d-1)), check.attributes = F) # residuals of regression and demeaned regression are the same
[1] TRUE
$\endgroup$
3
  • $\begingroup$ Thank you very much Christoph Hank for your detailed proof ! $\endgroup$
    – franzfranz
    Commented May 29, 2018 at 13:27
  • $\begingroup$ If you change the model to reg <- lm(y ~ X - 1) you will see there is a problem here. $\endgroup$
    – whuber
    Commented Nov 16, 2020 at 16:20
  • $\begingroup$ Indeed, the argument works when $X$ contains a constant - which my notation, as I realize, may not make as clear as it should - but the appeal to FWL does mention the need to have a constant. Also, when computing $R^2$ it is generally recommended to have a constant (vector in the column space of $X$) as it would otherwise not even be guaranteed to be nonnegative. $\endgroup$ Commented Nov 16, 2020 at 16:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.