Question on the Projection Theorem My question concerns the second sentence in the last paragraph i.e. 'The projection theorem guarantees that there is at least one solution of $\phi_n$ of (2.3.9)'. If a vector is uniquely defined by $\phi_n$ how is it that there can be more than one solution of $\phi_n$? Also, how does the singularity of $\Gamma_n$ imply that there are infinitely many solutions for $\phi_n$?

• The key is in the next sentence: "If $\Gamma_n$ is singular..." In that case, the vector $\phi_n$ is not uniquely defined. The reason is that you, in effect, have fewer than $n$ independent vectors that have been formed into $n$ linear combinations that make up $\Gamma_n$. In that case, your coefficients aren't unique either; if $x_1 + x_2 = x_3$, then $\beta_1, \beta_2$, and $\beta_3$ can take on lots of different values while retaining the same value of $x_1\beta_1 + x_2\beta_2 + x_3\beta_3$. – jbowman Apr 28 '18 at 16:39
• Thanks a lot @jbowman! So what does that mean geometrically? Surely there should only be one point in a linear subspace $\mathcal{M}$ where an element $x \in \mathcal{H}$ (where $\mathcal{H}$ is a Hilbert Space) can project to? – Vykta Wakandigara Apr 28 '18 at 16:49

Let's consider a simple example, one where $\Gamma_2$ has two columns $[\gamma_1, \gamma_2]$ that are identical: $\gamma_1 = \gamma_2$. In this case, any projection is necessarily on to the space spanned by, let us choose for concreteness, $\gamma_1$, although it in some sense appears to be onto a space spanned by two vectors.
Let us assume the best projection onto $\gamma_1$ is $\beta \gamma_1$. In the space $[\gamma_1, \gamma_2]$, though, we have that any $\beta_1, \beta_2$ such that $\beta_1\gamma_1 + \beta_2\gamma_2 = \beta\gamma_1$ will be an equally good projection. Since (in our special case) $\gamma_1 = \gamma_2$, all we need is that $\beta_1 + \beta_2 = \beta$, and there are an infinite number of $\beta_1, \beta_2$ combinations such that this will hold.
More generally, you're projecting onto a $k < n$ dimensional space, but you have $n$ vectors in that space to choose from when defining that projection. So there are clearly multiple ways of defining the projection, all of which are identical in terms of the end result.
• A plane cuts through the hypercube and you project onto the plane (when you think you're projecting onto the hypercube, if you don't know that $\Gamma$ is less than full rank.) Now, there is a line in the hypercube that is a) perpendicular to the plane, and b) points in the direction of the original point. Any point on this line is a) in the hypercube and b) indistinguishable from any other point, because they all get projected to the same point on the plane. At least that's how I envision it, but I'm not the best visual thinker around. – jbowman May 3 '18 at 18:16