# Least Square Fitted Vector Through SVD Equals to y

Elements of Statistical Learning, p 66

The SVD of the $$N \times p$$ matrix $$X$$ has the form $$X = UDV^T$$

Here $$U$$ and $$V$$ are $$N \times p$$ and $$p \times p$$ orthogonal matrices, with the columns of $$U$$ spanning the column space of $$X$$, and the columns of $$V$$ spanning the row space.

$$D$$ is a $$p \times p$$ diagonal matrix, with diagonal entries $$d_1 \ge d_2 \ge ... \ge d_p \ge 0$$ called the singular values of $$X$$. If one or more values $$d_j = 0$$, $$X$$ is singular.

Using the singular value decomposition we can write the least squares fitted vector as :

$$X \hat{\beta_{ls}} = X(𝑋^T𝑋)^{−1}𝑋^Ty = UU^Ty$$

The demonstration is available here p 266, and makes total sense.

Yet, as $$U$$ is orthogonal, it seems to me that it implies that $$UU^Ty=Iy=y$$, and thus that $$X\hat{\beta_{ls}}=y$$, which makes not sense. Indeed, it would mean that the the residual vector $$y-X\hat{\beta_{ls}}$$ is null. But this should only true in the specific case where $$y$$ can be expressed as a linear combination of the columns of $$X$$.

It seems to me that i'm totally missing something but i don't know what it is !

• I just updated with an example to show why it's not possible – jld Mar 19 '20 at 23:33
• Please visit stats.stackexchange.com/help/merging-accounts to merge your accounts, Tim, so you can edit your question freely. – whuber Mar 20 '20 at 1:53

$$U$$ has orthogonal columns but it's not square and isn't an orthogonal matrix. $$UU^T \neq I$$ in general since it only has rank $$p$$ and $$UU^T$$ involves dot products of the rows which don't have to be orthogonal when $$p. If $$U$$ was square then it would be the case that $$UU^T = I$$ which makes sense since then $$\hat y = y$$ since $$X$$ is a bijection.

I'm updating to add this example of why this isn't possible if $$n > p$$.

Let $$U = \left(\begin{array}{cc} a & b\\ c&d \\ e&f\end{array}\right).$$

Suppose $$U^TU = I_2$$ and $$UU^T = I_3$$. These turn into the following 9 constraints: $$a^2+c^2+e^2 = b^2+d^2+f^2 = 1 \hspace{2cm}(1)\\ ab + cd + ef = 0 \\ a^2 + b^2 = c^2 + d^2 = e^2 + f^2 = 1 \hspace{2cm}(2)\\ ac + bd = ae + bf = ce + df = 0$$ But we only have 6 variables, and as it turns out these are not satisfiable. To see this, note that we have $$a^2 + b^2 + c^2 + d^2 + e^2 +f^2 = 2$$ from adding the constraints in (1) together, but adding the constraints in line (2) together gives $$a^2 + b^2 + c^2 + d^2 + e^2 +f^2 = 3$$ which is impossible. Thus unless $$n=p$$ we don't have enough freedom to make $$U^TU = I$$ while also having $$UU^T = I$$. Since $$U^TU = I$$ is guaranteed by construction, the result is that $$UU^T \neq I$$ unless $$n=p$$.

I think interpreting this can also help with this. I like to interpret $$\hat y = UU^Ty$$ as follows: we can expand $$U$$ to an $$n\times n$$ matrix $$\tilde U = (U \mid U_\perp)$$ that actually does contain an orthonormal basis for $$\mathbb R^n$$ (this can be done via Gram-Schmidt). In this basis we'll have $$y = \tilde U \tilde z$$ where $$\tilde z \in \mathbb R^n$$ is $$y$$'s coordinate w.r.t. $$\tilde U$$. I'll partition $$\tilde z = {z \choose z_\perp}$$ where $$z \in \mathbb R^p$$ gives the coordinates for the basis vectors in $$U$$, and $$z_\perp$$ analogously gives the coordinates for the basis vectors in $$U_\perp$$.

This means $$y = \tilde U \tilde z = Uz + U_\perp z_\perp$$ so $$U^Ty = U^TUz + U^TU_\perp z_\perp = z$$ therefore $$U^Ty$$ picks out the coordinates of $$y$$ w.r.t. just the basis vectors in $$U$$. And then $$UU^Ty = Uz$$ gives the corresponding column space element in $$\mathbb R^n$$ with just those coordinates. So it's like doing a change of basis but only keeping the coordinates from $$U$$, and then getting the column space vector specified by those coordinates.

This reflects the fact that when we're talking about a $$p$$-dimensional subspace in $$\mathbb R^n$$, we can think of the actual vectors in $$\mathbb R^n$$, which are like $$X\beta$$, or we can think about the coordinates indexing the column space which is like $$\beta$$.

It says that $$U \in \mathbb{R}^{n \times p}$$ so the columns are orthogonal but the matrix is not necessarily orthogonal in the sense that $$UU^T =I$$ ($$\dim(UU^T) =p$$)