# In a linear regression, trying to show $R^2 = r_{xy}^2$ using projections / geometric intuition

In a linear regression $$Y = X\beta + \varepsilon,$$ I define two (standard) projection matrices. The projection matrix into subspace spanned by columns of the design matrix $$X$$: $$H := X(X^\top X)^{-1} X^\top,$$ and projection into the one dimensional subspace spanned by vector $$(1,\ldots, 1)$$: $$H_0 := \frac{1}{n}\mathbf{1} \mathbf{1}^\top.$$ (Note, one of columns of $$X$$, by convention, is a vector $$(1,\ldots, 1)$$, so we must have $$HH_0 = H_0$$).

According to my calculations (based on $$R^2 = r_{xy}^2$$, please, see below), it must be true that:

$$\|X^\top(I - H_0)Y\|^2= \|(H-H_0)Y\| \|(I - H_0)X\| \tag{1}$$

which I find a bit weird, it reminds me of Cauchy-Schwarz, but I couldn't decipher it in this way.

My question:

Is there an easy way (e.g., geometric, or inner product interpretation) to see why $$(1)$$ must be true?

The details are below.

Note Here I've asked this question on Mathematics Stackexchange, now I think the question might be on the linear algebra side, so I've decided to post it there as well. If I make progress I'll leave out one question only to avoid duplicates.

Details:

With the above projection matrices, $$H, H_0$$ define the standard quantities associated with a linear regression:

\begin{align} S_{YY} &:= \sum_{i=1}^n(y_i - \bar{y})^2 = \|(I - H_0)Y\|^2\,, \\ S_{XX} &:= \sum_{i=1}^n(x_i - \bar{x})^2 = \|(I - H_0)X\|^2\,, \\ S_{XY} &:= \sum_{i=1}^n(x_i - \bar{x})(y_i - \bar{y}) = \|X^\top(I - H_0)Y\|^2 = \|Y^\top(I - H_0)X\|^2\,,\\ R_{SS} &:= \sum_{i=1}^n(y_i - \hat{y}_i)^2 = \|(I-H)Y\|^2\,,\\ SS_{reg} &:= \sum_{i=1}^n(\hat{y}_i - \bar{\hat{y}_i})^2 = \sum_{i=1}^n(\hat{y}_i - \bar{{y}_i})^2 = \|(H-H_0)Y\|^2\,. \end{align}

Now, on the one hand, $$R^2:= \frac{\sum_{i=1}^n(\hat{y}_i - \bar{\hat{y}_i})^2 }{\sum_{i=1}^n(y_i - \bar{y})^2} = \frac{SS_{reg} }{S_{YY}} = \frac{\|(H-H_0)Y\|^2}{\|(I - H_0)Y\|^2},$$

and on the other hand

$$r^2_{xy}:= \frac{(\sum_{i=1}^n(x_i -\bar{x})(y_i -\bar{y}))^2}{\sum_{i=1}^n(x_i -\bar{x})^2\sum_{i=1}^n(y_i -\bar{y})^2} = \frac{S_{XY}^2}{S_{XX}S_{YY}} = \frac{ \|X^\top(I - H_0)Y\|^4}{\|(I - H_0)X\|^2\, \|(I - H_0)Y\|^2}.$$

It is a well known fact that the square of sample correlation coefficient and R squared are equal, $$r_{xy}^2 = R^2$$, which yields that

$$\frac{ \|X^\top(I - H_0)Y\|^4}{\|(I - H_0)X\|^2\, \|(I - H_0)Y\|^2} = \frac{\|(H-H_0)Y\|^2}{\|(I - H_0)Y\|^2}.$$ Or equivalently $$\|X^\top(I - H_0)Y\|^2= \|(H-H_0)Y\| \|(I - H_0)X\|.$$ The last expression looks weird, it remindes me of Cauchy-Schwarz, but I was not able to "decipher" it in this way, is there an easy why to see why $$(1)$$ must be true?

Would appreaciate any help.

I'm a little confused by the notation $$X$$. From the design matrix expression, it seems $$X$$ contains both columns of $$x$$ and the intercept. Then in (1) and Details, you seem to use it just for the vector of $$x$$. However, I think I get your question.

Let $$\tilde{Y} = (I - H_0) Y = Y - \mathbf{1}_n \bar{Y}$$, which is $$Y$$ with its mean removed. Similarly, $$\tilde{X} = (I - H_0)X$$ is also centered $$X$$.

It's easy to check that $$(I - H_0)(I - H_0) = I - H_0$$. The LHS of (1) can be simplified to

$$\|X^\top(I - H_0)Y\|^2 = [(I - H_0)X]^T (I - H_0) Y = \tilde{X}^T \tilde{Y}$$

You can also check that $$H - H_0 = (H - H_0)(I - H_0)$$. And this makes $$(H - H_0)Y = (H - H_0)(I - H_0)Y = (H - H_0)\tilde{Y}$$

Note that $$\tilde{X}$$ is now perpendicular to $$\mathbf{1}_n$$, therefore the projection of $$\tilde{Y}$$ onto $$\text{span}\{X, \mathbf{1}_n\}$$, $$Proj_{\{X, \mathbf{1}_n\}}(Y)$$, is just the sum of $$\text{Proj}_{\mathbf{1}_n} (\tilde{Y})$$ and $$\text{Proj}_{\tilde{X}} (\tilde{Y})$$, or in mathematical terms $$H\tilde{Y} = H_0\tilde{Y} + \text{Proj}_{\tilde{X}} (\tilde{Y})$$ $$\Rightarrow \text{Proj}_{\tilde{X}} (\tilde{Y}) = (H - H_0)\tilde{Y}$$

Finally, the equation (1) boils down to:

$$\tilde{X}^T \tilde{Y} = \|\text{Proj}_{\tilde{X}} (\tilde{Y})\| \|\tilde{X}\|$$

Since $$\tilde{Y} - \text{Proj}_{\tilde{X}}(\tilde{Y})$$ is perpendicular to $$\tilde{X}$$ by definition of projection, the above equation really is

$$\tilde{X}^T \text{Proj}_{\tilde{X}}(\tilde{Y}) = \|\text{Proj}_{\tilde{X}} (\tilde{Y})\| \|\tilde{X}\|$$

which is exactly Cauchy (in)equality. The equality is achieved because $$\tilde{X}$$ and $$\text{Proj}_{\tilde{X}}(\tilde{Y})$$ only differs by a constant factor.