1
$\begingroup$

enter image description here

Screenshot from page 80 of the textbook "Introduction to Linear Regression Analysis" fifth edition by Douglas C. Montgomery

Let $X$ be $n \times p$, $y$ and $\hat{y}$ be $n \times 1,$ and $\hat{\beta}$ be $p \times 1$ matrix in the multiple linear regression model. From the matrix calculation, we can easily find $\hat{\beta} = (X'X)^{-1}X'y$ and $\hat{y}=X\hat{\beta}$. However, in the later part of this chapter, the estimation of $\sigma^{2}$, while calculating the residual sum of squares, the textbook says $X'X\hat{\beta} = X'y$ which indicates $X\hat{\beta}=y$. But I don't understand why it is not $\hat{y}$. Can anyone answer this question?

$\endgroup$
0

1 Answer 1

3
$\begingroup$

$X'X\hat{\beta} = X'X \left((X'X)^{-1}X'y\right)=\left(X'X (X'X)^{-1}\right)X'y=X'y$

This does not imply that $X\hat{\beta}=y$ though. In algebra, the statement $CA=CB$ only implies that $A=B$ if $C$ is invertible, and, in this case, $C=X'$ is not even (necessarily) a square matrix.

What it does imply, however, is that $X'y = X'\hat y$, which is true in general, since $X'e = X'y-X'X\hat \beta=0$.


Addendum:

If $X$ is a square matrix with full-rank ($n=p$), then $y=\hat y$.

$\endgroup$
3
  • 1
    $\begingroup$ Thank you for answering this question clearly. By the way, is it the typo in the first line that you wrote $X'y$ outside the bracket from $X'X((X'X)^{-1}X'y)X'y$? Would it be $X'X((X'X)^{-1}X'y)$? $\endgroup$
    – user375779
    Commented Apr 13, 2023 at 7:02
  • $\begingroup$ Yes, sorry about that @user375779 :) $\endgroup$
    – Firebug
    Commented Apr 13, 2023 at 7:03
  • 2
    $\begingroup$ For a specific example, consider the case of a regression of $y$ on a constant, $X=\mathbf{1}$. Then $\hat\beta=\bar y$ and thus $X\hat\beta=\mathbf{1}\bar{y}$, the sample mean replicated into a vector of ones, which would only be equal to $y$ if $y$ always took the same value. $\endgroup$ Commented Apr 13, 2023 at 12:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.