In a linear regression, trying to show $R^2 = r_{xy}^2$ using projections / geometric intuition

Question

In a linear regression $$ Y = X\beta + \varepsilon, $$ I define two (standard) projection matrices. The projection matrix into subspace spanned by columns of the design matrix $X$: $$ H := X(X^\top X)^{-1} X^\top, $$ and projection into the one dimensional subspace spanned by vector $(1,\ldots, 1)$: $$ H_0 := \frac{1}{n}\mathbf{1} \mathbf{1}^\top. $$ (Note, one of columns of $X$, by convention, is a vector $(1,\ldots, 1)$, so we must have $HH_0 = H_0$).

According to my calculations (based on $R^2 = r_{xy}^2$, please, see below), it must be true that:

$$ \|X^\top(I - H_0)Y\|^2= \|(H-H_0)Y\| \|(I - H_0)X\| \tag{1} $$

which I find a bit weird, it reminds me of Cauchy-Schwarz, but I couldn't decipher it in this way.

My question:

Is there an easy way (e.g., geometric, or inner product interpretation) to see why $(1)$ must be true?

The details are below.

Note Here I've asked this question on Mathematics Stackexchange, now I think the question might be on the linear algebra side, so I've decided to post it there as well. If I make progress I'll leave out one question only to avoid duplicates.

---

---

Details:

With the above projection matrices, $H, H_0$ define the standard quantities associated with a linear regression:

\begin{align} S_{YY} &:= \sum_{i=1}^n(y_i - \bar{y})^2 = \|(I - H_0)Y\|^2\,, \\ S_{XX} &:= \sum_{i=1}^n(x_i - \bar{x})^2 = \|(I - H_0)X\|^2\,, \\ S_{XY} &:= \sum_{i=1}^n(x_i - \bar{x})(y_i - \bar{y}) = \|X^\top(I - H_0)Y\|^2 = \|Y^\top(I - H_0)X\|^2\,,\\ R_{SS} &:= \sum_{i=1}^n(y_i - \hat{y}_i)^2 = \|(I-H)Y\|^2\,,\\ SS_{reg} &:= \sum_{i=1}^n(\hat{y}_i - \bar{\hat{y}_i})^2 = \sum_{i=1}^n(\hat{y}_i - \bar{{y}_i})^2 = \|(H-H_0)Y\|^2\,. \end{align}

Now, on the one hand, $$ R^2:= \frac{\sum_{i=1}^n(\hat{y}_i - \bar{\hat{y}_i})^2 }{\sum_{i=1}^n(y_i - \bar{y})^2} = \frac{SS_{reg} }{S_{YY}} = \frac{\|(H-H_0)Y\|^2}{\|(I - H_0)Y\|^2}, $$

and on the other hand

$$ r^2_{xy}:= \frac{(\sum_{i=1}^n(x_i -\bar{x})(y_i -\bar{y}))^2}{\sum_{i=1}^n(x_i -\bar{x})^2\sum_{i=1}^n(y_i -\bar{y})^2} = \frac{S_{XY}^2}{S_{XX}S_{YY}} = \frac{ \|X^\top(I - H_0)Y\|^4}{\|(I - H_0)X\|^2\, \|(I - H_0)Y\|^2}. $$

It is a well known fact that the square of sample correlation coefficient and R squared are equal, $r_{xy}^2 = R^2$, which yields that

$$ \frac{ \|X^\top(I - H_0)Y\|^4}{\|(I - H_0)X\|^2\, \|(I - H_0)Y\|^2} = \frac{\|(H-H_0)Y\|^2}{\|(I - H_0)Y\|^2}. $$ Or equivalently $$ \|X^\top(I - H_0)Y\|^2= \|(H-H_0)Y\| \|(I - H_0)X\|. $$ The last expression looks weird, it remindes me of Cauchy-Schwarz, but I was not able to "decipher" it in this way, is there an easy why to see why $(1)$ must be true?

Would appreaciate any help.

Answer 1

I'm a little confused by the notation $X$. From the design matrix expression, it seems $X$ contains both columns of $x$ and the intercept. Then in (1) and Details, you seem to use it just for the vector of $x$. However, I think I get your question.

Let $\tilde{Y} = (I - H_0) Y = Y - \mathbf{1}_n \bar{Y}$, which is $Y$ with its mean removed. Similarly, $\tilde{X} = (I - H_0)X$ is also centered $X$.

It's easy to check that $(I - H_0)(I - H_0) = I - H_0$. The LHS of (1) can be simplified to

$$\|X^\top(I - H_0)Y\|^2 = [(I - H_0)X]^T (I - H_0) Y = \tilde{X}^T \tilde{Y}$$

You can also check that $H - H_0 = (H - H_0)(I - H_0)$. And this makes $$(H - H_0)Y = (H - H_0)(I - H_0)Y = (H - H_0)\tilde{Y}$$

Note that $\tilde{X}$ is now perpendicular to $\mathbf{1}_n$, therefore the projection of $\tilde{Y}$ onto $\text{span}\{X, \mathbf{1}_n\}$, $Proj_{\{X, \mathbf{1}_n\}}(Y)$, is just the sum of $\text{Proj}_{\mathbf{1}_n} (\tilde{Y})$ and $\text{Proj}_{\tilde{X}} (\tilde{Y})$, or in mathematical terms $$H\tilde{Y} = H_0\tilde{Y} + \text{Proj}_{\tilde{X}} (\tilde{Y})$$ $$\Rightarrow \text{Proj}_{\tilde{X}} (\tilde{Y}) = (H - H_0)\tilde{Y}$$

Finally, the equation (1) boils down to:

$$\tilde{X}^T \tilde{Y} = \|\text{Proj}_{\tilde{X}} (\tilde{Y})\| \|\tilde{X}\|$$

Since $\tilde{Y} - \text{Proj}_{\tilde{X}}(\tilde{Y})$ is perpendicular to $\tilde{X}$ by definition of projection, the above equation really is

$$\tilde{X}^T \text{Proj}_{\tilde{X}}(\tilde{Y}) = \|\text{Proj}_{\tilde{X}} (\tilde{Y})\| \|\tilde{X}\|$$

which is exactly Cauchy (in)equality. The equality is achieved because $\tilde{X}$ and $\text{Proj}_{\tilde{X}}(\tilde{Y})$ only differs by a constant factor.