4
$\begingroup$

Can someone provide a proof why the coefficient of determination given by

$$R^2:=1-\dfrac{||y-\hat{y}||_2^2}{||y-\overline{y}||_2^2}$$

in a multiple linear regression setting

$$y=X\beta+\epsilon$$

is invariant when standardizing (centering and dividing by the standard deviation) $X$ and $y$?

$\endgroup$
  • $\begingroup$ Is this homework or self-study? $\endgroup$ – cherub May 29 '18 at 11:56
  • 2
    $\begingroup$ dear cherub its not related to a homework task.I often read that statement in the literature, but have never seen a proof for this, so I was interested in the proof. best $\endgroup$ – franzfranz May 29 '18 at 12:29
2
$\begingroup$

Denote $$\tilde{y}=\frac{y-\bar{y}}{\sigma_y}=\frac{Hy}{\sigma_y}$$ and $$\tilde{X}=HX\Sigma_x$$ the scaled variables, where $H=I-\iota\iota'/n$ denotes the demeaning matrix and $\Sigma_x$ is a diagonal matrix with $1/\sigma_{x_j}$ on the main diagonal, $j=1,\ldots,K$.

Step 1: In the denominator of the $R^2$ of the regression of the scaled variables, we have $$||\tilde{y}-\overline{\tilde{y}}||_2^2=(\tilde{y}-\bar{\tilde{y}})'(\tilde{y}-\bar{\tilde{y}})=(y-\bar{y})'(y-\bar{y})/\sigma_y^2,$$ as demeaning the demeaned variable again will evidently do nothing.

Step 2: The numerator gives the sum of squared residuals of the regression. For the standard regression of $y$ on $X$, this well-known expression reads, in matrix notation, as $$y'(I-X(X'X)^{-1}X')y.\tag{1}$$

For the regression of the scaled variables, i.e., $\tilde{y}$ on $\tilde{X}$, we thus have $$ ||\tilde{y}-\hat{\tilde{y}}||_2^2=\frac{1}{\sigma_y^2}y'H[I-HX\Sigma_x(\Sigma_xX'HX\Sigma_x)^{-1}\Sigma_xX'H]Hy, $$ where it was used that $H$ is symmetric and idempotent and that $\Sigma_x$ is symmetric. Next, since $\Sigma_x$ is also invertible, the expression in square brackets simplifies to $$ I-HX(X'HX)^{-1}X'H, $$ as $(\Sigma_xX'HX\Sigma_x)^{-1}=\Sigma_x^{-1}(X'HX)^{-1}\Sigma_x^{-1}$. (Incidentally, this argument reveals that scaling $HX$ by any invertible matrix $A$ rather than just $\Sigma_x$ would do the trick, as would dividing $Hy$ by any $c\neq0$.)

Thus, we obtain $$ ||\tilde{y}-\hat{\tilde{y}}||_2^2=\frac{1}{\sigma_y^2}y'H[I-HX(X'HX)^{-1}X'H]Hy, $$ Here, in analogy to (1), $$ y'H[I-HX(X'HX)^{-1}X'H]Hy $$ denotes the sum of squared residuals of a regression of $Hy$ on $HX$. By the FWL theorem, these residuals are the same as those of a regression of $y$ on $X$ (provided $X$ contains a constant, or at least has columns that can be combined into a constant), i.e. $$ ||y-\hat{y}||_2^2=y'H[I-HX(X'HX)^{-1}X'H]Hy. $$

Hence, \begin{eqnarray*} R^2_{scaled}&:=&1-\dfrac{||\tilde{y}-\hat{\tilde{y}}||_2^2}{||\tilde{y}-\overline{\tilde{y}}||_2^2}\\ &=&1-\dfrac{||y-\hat{y}||_2^2/\sigma_y^2}{||y-\overline{y}||_2^2/\sigma_y^2}\\ &=&R^2 \end{eqnarray*}

For those who, like me, like numerical illustrations of formal results:

y <- runif(10)
X <- runif(10)

reg <- lm(y~X)

y.s <- scale(y)         # scaled y
X.s <- scale(X)         # scaled X
scale.reg <- lm(y.s~X.s)

y.d <- scale(y,scale=F) # demeaned y
X.d <- scale(X,scale=F) # demeaned X

> all.equal(summary(reg)$r.squared, summary(scale.reg)$r.squared)   # R^2s are the same
[1] TRUE

> all.equal(sum(resid(reg)^2)/sd(y)^2, sum(resid(scale.reg)^2))     # comparison numerators
[1] TRUE

> all.equal(sum((y-mean(y))^2)/sd(y)^2, sum((y.s-mean(y.s))^2))     # comparison denominators
[1] TRUE

> all.equal(resid(reg), resid(lm(y.d~X.d-1)), check.attributes = F) # residuals of regression and demeaned regression are the same
[1] TRUE
$\endgroup$
  • $\begingroup$ Thank you very much Christoph Hank for your detailed proof ! $\endgroup$ – franzfranz May 29 '18 at 13:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.