Please I know the least squares solution for $\hat\beta = (X^TX)^{-1}X^Ty$ but I don't know how they were able to get
$X\hat\beta= X(X^TX)^{-1}X^Ty = UU^Ty$
These are the steps I followed :
$X\hat\beta = U\sum V^T(V\sum^T U^TU\sum V^T)^{-1}V\sum^T U^Ty \quad (1)$
$X\hat\beta = U\sum V^TV\sum^T U^Ty \quad(2)$
$X\hat\beta = UU^Ty \quad(3)$
Please from $(1)$ , why is $U^TU = I$ and $UU^Ty \neq I$?
From the the definition of SVD, $U$ and $V$ are orthogonal matrices. So the products $U^TU$, $V^TV$ are identity matrices.
\begin{cases} X = USV^T\\ X\hat\beta = X(X^TX)^{-1}X^Ty \end{cases}
So
$$X\hat\beta = (USV^T)((USV^T)^T(USV^T))^{-1}(USV^T)^Ty\\ =USV^T(VS\color{red}{U^TU}SV^T)^{-1}VSU^Ty\\ =USV^T(VS^2V^T)^{-1}VSU^Ty\\ =US\color{red}{V^TV}S^{-2}\color{red}{V^TV}SU^Ty\\ =U\color{red}{SS^{-2}S}U^Ty\\ =UU^Ty\\$$
On why $UU^T$ is not necessarily $\mathbb I$, this stems from the definition of the SVD. A compelling argument can be given by the last line of our derivation, though: $X\hat\beta$ is not necessarily equal to $y$, thus $UU^T$ is not necessarily the identity matrix.
