9
$\begingroup$

In linear regression I have come across a delightful result that if we fit the model

$$E[Y] = \beta_1 X_1 + \beta_2 X_2 + c,$$

then, if we standardize and centre the $Y$, $X_1$ and $X_2$ data,

$$R^2 = \mathrm{Cor}(Y,X_1) \beta_1 + \mathrm{Cor}(Y, X_2) \beta_2.$$

This feels to me like a 2 variable version of $R^2 = \mathrm{Cor}(Y,X)^2$ for $y=mx+c$ regression, which is pleasing.

But the only proof I know is not in anyway constructive or insightful (see below), and yet to look at it it feels like it should be readily understandable.

Example thoughts:

  • The $\beta_1$ and $\beta_2$ parameters give us the 'proportion' of $X_1$ and $X_2$ in $Y$, and so we are taking respective proportions of their correlations...
  • The $\beta$s are partial correlations, $R^2$ is the squared multiple correlation... correlations multiplied by partial correlations...
  • If we orthogonalize first then the $\beta$s will be $\mathrm{Cov}/\mathrm{Var}$... does this result make some geometric sense?

None of these threads seem to lead anywhere for me. Can anyone provide a clear explanation of how to understand this result.


Unsatisfying Proof

\begin{equation} R^2 = \frac{SS_{reg}}{SS_{Tot}} = \frac{SS_{reg}}{N} = \langle(\beta_1 X_1 + \beta_2 X_2)^2\rangle \\= \langle\beta_1^2 X_1^2\rangle + \langle\beta_2^2 X_2^2\rangle + 2\langle\beta_1\beta_2X_1X_2\rangle \end{equation}

and

\begin{equation} \mathrm{Cor}(Y,X_1) \beta_1 + \mathrm{Cor}(Y, X_2) \beta_2 = \langle YX_1\rangle\beta_1 + \langle Y X_2\rangle \beta_2\\ =\langle \beta_1 X_1^2 + \beta_2 X_1 X_2\rangle \beta_1 + \langle \beta_1 X_1 X_2 + \beta_2 X_2^2\rangle \beta_2\\ =\langle \beta_1^2 X_1^2\rangle + \langle \beta_2^2 X_2^2 \rangle + 2\langle \beta_1 \beta_2 X_1 X_2\rangle \end{equation}

QED.

$\endgroup$
  • $\begingroup$ You must be using standardized variables, for otherwise your formula for $R^2$ is not guaranteed to lie between $0$ and $1$. Although this assumption comes out in your proof, it would help to make it explicit at the outset. I am puzzled over what you are really doing, too: your $R^2$ clearly is a function of the model alone--having nothing to do with the data--yet you start out mentioning that you have "fit" the model to something. $\endgroup$ – whuber Nov 20 '14 at 22:25
  • $\begingroup$ Doesn't your top result only hold if X1 & X2 are perfectly uncorrelated? $\endgroup$ – gung Nov 20 '14 at 22:34
  • $\begingroup$ @gung I don't think so - proof at bottom seems to say it works regardless. This result surprises me too, hence wanting a "clear understanding proof" $\endgroup$ – Corone Nov 20 '14 at 22:35
  • $\begingroup$ @whuber I'm not sure what you mean by "function of the model alone"? I simply mean the $R^2$ for simple OLS with two predicter variables. I.e. this is the 2 variable version of $R^2 = Cor(Y,X)^2$ $\endgroup$ – Corone Nov 20 '14 at 22:41
  • $\begingroup$ I cannot tell whether your $\beta_i$ are the parameters or the estimates. $\endgroup$ – whuber Nov 20 '14 at 22:53
9
$\begingroup$

The hat matrix is idempotent.

(This is a linear-algebraic way of stating that OLS is an orthogonal projection of the response vector onto the space spanned by the variables.)


Recall that by definition

$$R^2 = \frac{ESS}{TSS}$$

where

$$ESS = (\hat Y)^\prime \hat Y$$

is the sum of squares of the (centered) predicted values and

$$TSS = Y^\prime Y$$

is the sum of squares of the (centered) response values. Standardizing $Y$ beforehand to unit variance also implies

$$TSS = Y^\prime Y = n.$$

Recall, too, that the estimated coefficients are given by

$$\hat\beta = (X^\prime X)^{-} X^\prime Y,$$

whence

$$\hat Y = X \hat \beta = X (X^\prime X)^{-} X^\prime Y = H Y$$

where $H$ is the "hat matrix" effecting the projection of $Y$ onto its least squares fit $\hat Y$. It is symmetric (which is obvious from its very form) and idempotent. Here is a proof of the latter for those unfamiliar with this result. It's just shuffling parentheses around:

$$\eqalign{H^\prime H = H H &=\left( X (X^\prime X)^{-} X^\prime\right)\left(X (X^\prime X)^{-} X^\prime \right) \\ &= X (X^\prime X)^{-} \left(X^\prime X \right) (X^\prime X)^{-} X^\prime \\ &= X (X^\prime X)^{-} X^\prime = H. }$$

Therefore

$$R^2 = \frac{ESS}{TSS} = \frac{1}{n} (\hat Y)^\prime \hat Y = \frac{1}{n}Y^\prime H^\prime H Y = \frac{1}{n}Y^\prime H Y = \left(\frac{1}{n}Y^\prime X\right) \hat \beta.$$

The crucial move in the middle used the idempotence of the hat matrix. The right hand side is your magical formula because $\frac{1}{n}Y^\prime X$ is the (row) vector of correlation coefficients between $Y$ and the columns of $X$.

$\endgroup$
  • $\begingroup$ (+1) Very nice write-up. But why ^{-} instead of ^{-1} everywhere? $\endgroup$ – amoeba Nov 20 '14 at 23:32
  • $\begingroup$ @amoeba It's a generalized inverse, put there to handle the cases where $X^\prime X$ may be singular. $\endgroup$ – whuber Nov 20 '14 at 23:33
  • 3
    $\begingroup$ @amoeba Penrose, in his original paper (A Generalized Inverse for Matrices, 1954) used the notation $A^\dagger$. I like neither that nor the $A^{+}$ notation because they are too easily confused with conjugates, transposes, or conjugate transposes, whereas the $A^{-}$ notation is so suggestive of an inverse the casual reader can get away with thinking of it as $A^{-1}$ if they like. You're just too good a reader--but thanks for noticing. $\endgroup$ – whuber Nov 20 '14 at 23:56
  • 1
    $\begingroup$ Interesting and compelling motivation, but may I ask if this notation is something that is occasionally used elsewhere or is it your own invention? $\endgroup$ – amoeba Nov 21 '14 at 0:00
  • 5
    $\begingroup$ @amoeba: Yes, this notation appears elsewhere, including in the classical texts by Graybill on the linear model. $\endgroup$ – cardinal Nov 21 '14 at 1:07
4
$\begingroup$

The following three formulas are well known, they are found in many books on linear regression. It is not difficult to derive them.

$\beta_1= \frac {r_{YX_1}-r_{YX_2}r_{X_1X_2}} {\sqrt{1-r_{X_1X_2}^2}}$

$\beta_2= \frac {r_{YX_2}-r_{YX_1}r_{X_1X_2}} {\sqrt{1-r_{X_1X_2}^2}}$

$R^2= \frac {r_{YX_1}^2+r_{YX_2}^2-2 r_{YX_1}r_{YX_2}r_{X_1X_2}} {\sqrt{1-r_{X_1X_2}^2}}$

If you substitute the two betas into your equation $R^2 = r_{YX_1} \beta_1 + r_{YX_2} \beta_2$, you will get the above formula for R-square.


Here is a geometric "insight". Below are two pictures showing regression of $Y$ by $X_1$ and $X_2$. This kind of representation is known as variables-as-vectors in subject space (please read what it is about). The pictures are drawn after all the three variables were centered, and so (1) every vector's length = st. deviation of the respective variable, and (2) angle (its cosine) between every two vectors = correlation between the respective variables.

enter image description here

$\hat{Y}$ is the regression prediction (orthogonal projection of $Y$ onto "plane X"); $e$ is the error term; $cos \angle{Y \hat{Y}}={|\hat Y|}/|Y|$, multiple correlation coefficient.

The left picture depicts skew coordinates of $\hat{Y}$ on variables $X_1$ and $X_2$. We know that such coordinates relate the regression coefficients. Namely, the coordinates are: $b_1|X_1|=b_1\sigma_{X_1}$ and $b_2|X_2|=b_2\sigma_{X_2}$.

And the right picture shows corresponding perpendicular coordinates. We know that such coordinates relate the zero order correlation coefficients (these are cosines of orthogonal projections). If $r_1$ is the correlation between $Y$ and $X_1$ and $r_1^*$ is the correlation between $\hat Y$ and $X_1$ then the coordinate is $r_1|Y|=r_1\sigma_{Y} = r_1^*|\hat{Y}|=r_1^*\sigma_{\hat{Y}}$. Likewise for the other coordinate, $r_2|Y|=r_2\sigma_{Y} = r_2^*|\hat{Y}|=r_2^*\sigma_{\hat{Y}}$.

So far it were general explanations of linear regression vector representation. Now we turn for the task to show how it may lead to $R^2 = r_1 \beta_1 + r_2 \beta_2$.

First of all, recall that in their question @Corone put forward the condition that the expression is true when all the three variables are standardized, that is, not just centered but also scaled to variance 1. Then (i.e. implying $|X_1|=|X_2|=|Y|=1$ to be the "working parts" of the vectors) we have coordinates equal to: $b_1|X_1|=\beta_1$; $b_2|X_2|=\beta_2$; $r_1|Y|=r_1$; $r_2|Y|=r_2$; as well as $R=|\hat Y|/|Y|=|\hat Y|$. Redraw, under these conditions, just the "plane X" of the pictures above:

enter image description here

On the picture, we have a pair of perpendicular coordinates and a pair of skew coordinates, of the same vector $\hat Y$ of length $R$. There exist a general rule to obtain perpendicular coordinates from skew ones (or back): $\bf P = S C$, where $\bf P$ is points X axes matrix of perpendicular ones; $\bf S$ is the same sized matrix of skew ones; and $\bf C$ are the axes X axes symmetric matrix of angles (cosines) between the nonorthogonal axes.

$X_1$ and $X_2$ are the axes in our case, with $r_{12}$ being the cosine between them. So, $r_1 = \beta_1 + \beta_2 r_{12}$ and $r_2 = \beta_1 r_{12} + \beta_2$.

Substitute these $r$s expressed via $\beta$s in the @Corone's statement $R^2 = r_1 \beta_1 + r_2 \beta_2$, and you'll get that $R^2 = \beta_1^2 + \beta_2^2 + 2\beta_1\beta_2r_{12}$, - which is true, because it is exactly how a diagonal of a parallelogram (tinted on the picture) is expressed via its adjacent sides (quantity $\beta_1\beta_2r_{12}$ being the scalar product).

This same thing is true for any number of predictors X. Unfortunately, it is impossible to draw the alike pictures with many predictors.

$\endgroup$
  • 1
    $\begingroup$ +1 nice to see it constructed this way as well, but this doesn't add as much insight compared to whuber's answer $\endgroup$ – Corone Nov 21 '14 at 10:58
  • 2
    $\begingroup$ @Corone, I added some "insight" which you might take. $\endgroup$ – ttnphns Nov 22 '14 at 13:29
  • 1
    $\begingroup$ +1 Really cool (after the update). I thought that invoking "general rule" of converting between coordinates is a bit of an overkill (and for me was only confusing); to see that e.g. $r_1 = \beta_1 + \beta_2 r_{12}$ one only needs to remember the definition of cosine and look at one of the right triangles. $\endgroup$ – amoeba Nov 22 '14 at 15:31
  • $\begingroup$ Really cool edit, switched accepted. $\endgroup$ – Corone Nov 22 '14 at 19:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.