5
$\begingroup$

In a linear regression $$ Y = X\beta + \varepsilon, $$ I define two (standard) projection matrices. The projection matrix into subspace spanned by columns of the design matrix $X$: $$ H := X(X^\top X)^{-1} X^\top, $$ and projection into the one dimensional subspace spanned by vector $(1,\ldots, 1)$: $$ H_0 := \frac{1}{n}\mathbf{1} \mathbf{1}^\top. $$ (Note, one of columns of $X$, by convention, is a vector $(1,\ldots, 1)$, so we must have $HH_0 = H_0$).

According to my calculations (based on $R^2 = r_{xy}^2$, please, see below), it must be true that:

$$ \|X^\top(I - H_0)Y\|^2= \|(H-H_0)Y\| \|(I - H_0)X\| \tag{1} $$

which I find a bit weird, it reminds me of Cauchy-Schwarz, but I couldn't decipher it in this way.

My question:

Is there an easy way (e.g., geometric, or inner product interpretation) to see why $(1)$ must be true?

The details are below.

Note Here I've asked this question on Mathematics Stackexchange, now I think the question might be on the linear algebra side, so I've decided to post it there as well. If I make progress I'll leave out one question only to avoid duplicates.



Details:

With the above projection matrices, $H, H_0$ define the standard quantities associated with a linear regression:

\begin{align} S_{YY} &:= \sum_{i=1}^n(y_i - \bar{y})^2 = \|(I - H_0)Y\|^2\,, \\ S_{XX} &:= \sum_{i=1}^n(x_i - \bar{x})^2 = \|(I - H_0)X\|^2\,, \\ S_{XY} &:= \sum_{i=1}^n(x_i - \bar{x})(y_i - \bar{y}) = \|X^\top(I - H_0)Y\|^2 = \|Y^\top(I - H_0)X\|^2\,,\\ R_{SS} &:= \sum_{i=1}^n(y_i - \hat{y}_i)^2 = \|(I-H)Y\|^2\,,\\ SS_{reg} &:= \sum_{i=1}^n(\hat{y}_i - \bar{\hat{y}_i})^2 = \sum_{i=1}^n(\hat{y}_i - \bar{{y}_i})^2 = \|(H-H_0)Y\|^2\,. \end{align}

Now, on the one hand, $$ R^2:= \frac{\sum_{i=1}^n(\hat{y}_i - \bar{\hat{y}_i})^2 }{\sum_{i=1}^n(y_i - \bar{y})^2} = \frac{SS_{reg} }{S_{YY}} = \frac{\|(H-H_0)Y\|^2}{\|(I - H_0)Y\|^2}, $$

and on the other hand

$$ r^2_{xy}:= \frac{(\sum_{i=1}^n(x_i -\bar{x})(y_i -\bar{y}))^2}{\sum_{i=1}^n(x_i -\bar{x})^2\sum_{i=1}^n(y_i -\bar{y})^2} = \frac{S_{XY}^2}{S_{XX}S_{YY}} = \frac{ \|X^\top(I - H_0)Y\|^4}{\|(I - H_0)X\|^2\, \|(I - H_0)Y\|^2}. $$

It is a well known fact that the square of sample correlation coefficient and R squared are equal, $r_{xy}^2 = R^2$, which yields that

$$ \frac{ \|X^\top(I - H_0)Y\|^4}{\|(I - H_0)X\|^2\, \|(I - H_0)Y\|^2} = \frac{\|(H-H_0)Y\|^2}{\|(I - H_0)Y\|^2}. $$ Or equivalently $$ \|X^\top(I - H_0)Y\|^2= \|(H-H_0)Y\| \|(I - H_0)X\|. $$ The last expression looks weird, it remindes me of Cauchy-Schwarz, but I was not able to "decipher" it in this way, is there an easy why to see why $(1)$ must be true?

Would appreaciate any help.

$\endgroup$
1
$\begingroup$

I'm a little confused by the notation $X$. From the design matrix expression, it seems $X$ contains both columns of $x$ and the intercept. Then in (1) and Details, you seem to use it just for the vector of $x$. However, I think I get your question.

Let $\tilde{Y} = (I - H_0) Y = Y - \mathbf{1}_n \bar{Y}$, which is $Y$ with its mean removed. Similarly, $\tilde{X} = (I - H_0)X$ is also centered $X$.

It's easy to check that $(I - H_0)(I - H_0) = I - H_0$. The LHS of (1) can be simplified to

$$\|X^\top(I - H_0)Y\|^2 = [(I - H_0)X]^T (I - H_0) Y = \tilde{X}^T \tilde{Y}$$

You can also check that $H - H_0 = (H - H_0)(I - H_0)$. And this makes $$(H - H_0)Y = (H - H_0)(I - H_0)Y = (H - H_0)\tilde{Y}$$

Note that $\tilde{X}$ is now perpendicular to $\mathbf{1}_n$, therefore the projection of $\tilde{Y}$ onto $\text{span}\{X, \mathbf{1}_n\}$, $Proj_{\{X, \mathbf{1}_n\}}(Y)$, is just the sum of $\text{Proj}_{\mathbf{1}_n} (\tilde{Y})$ and $\text{Proj}_{\tilde{X}} (\tilde{Y})$, or in mathematical terms $$H\tilde{Y} = H_0\tilde{Y} + \text{Proj}_{\tilde{X}} (\tilde{Y})$$ $$\Rightarrow \text{Proj}_{\tilde{X}} (\tilde{Y}) = (H - H_0)\tilde{Y}$$

Finally, the equation (1) boils down to:

$$\tilde{X}^T \tilde{Y} = \|\text{Proj}_{\tilde{X}} (\tilde{Y})\| \|\tilde{X}\|$$

Since $\tilde{Y} - \text{Proj}_{\tilde{X}}(\tilde{Y})$ is perpendicular to $\tilde{X}$ by definition of projection, the above equation really is

$$\tilde{X}^T \text{Proj}_{\tilde{X}}(\tilde{Y}) = \|\text{Proj}_{\tilde{X}} (\tilde{Y})\| \|\tilde{X}\|$$

which is exactly Cauchy (in)equality. The equality is achieved because $\tilde{X}$ and $\text{Proj}_{\tilde{X}}(\tilde{Y})$ only differs by a constant factor.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.