How does Linear Regression form a Subspace? In Elements of Statistical Learning, the authors note that Linear Regression can be rewritten from
$$ \hat{Y} = \hat{\beta}_0 + \sum_{j = 1}^p X_j\hat{\beta}_j$$
to
$$ \hat{Y} = X^T \hat{\beta} $$
by including $\hat{\beta}_0$ in the vector of coefficients and including a column of $1$s in $X$.  The author's then go on to note that $(X,\hat{Y})$ form a subspace in the $p+1$ dimensional input-output space.
I can't really see why this is true. I understand the axioms of Vector Spaces and Subspaces, but don't understand the argument the authors are making.  Could someone please shed some light?
 A: Interpretation
The sense in which "$(X,\hat Y)$ form a subspace" is that the $p+1$ columns of $X$ (let's call them $X_0, X_1, \ldots, X_{p}$), together with the vector $\hat Y$, generate a subspace of the space generated by the columns of $X$ alone.  This assertion embodies several subtleties:


*

*The dimension of this subspace need not be exactly $p+1$, even though $X$ has $p+1$ columns.  It can be as small as $1$ (which occurs when there are no regressors at all or when all regressors are identically zero).

*The columns of $X$ and $Y$ itself may be elements of infinite-dimensional vector spaces, such as spaces of functions--which include spaces of random variables.  This powerful generalization of linear regression comes to us for free, without any additional concepts or calculations being necessary.

*It doesn't matter what the field of scalars is.  Thus, this result holds even when you are allowing the $\hat \beta$ to be complex numbers.  A comparable result holds--taking care to preserve the order of multiplication in every instance--even for noncommutative fields, such as the Quaternions.

*The regressors determine the subspace, but the subspace corresponds to many different choices of regressors.  It is only the subspace that matters, because linear regression finds (the unique) projection from $Y$ into this subspace.  In solving any linear regression problem, therefore, we should think of the regressors solely in terms of the subspace they generate, rather than as individual vectors. This point of view simplifies discussions of identifiability.
What the assertion means
By definition, whenever $E\subset V$ is a subset of a vector space, the subspace it generates consists of all vectors that can be written as finite linear combinations of elements of $E$:
$$\langle E \rangle = \{\alpha_1 e_1 + \alpha_2 e_2 + \cdots \alpha_k e_k\mid e_1,\ldots,e_k\in E\}.$$
When $E\subset F$ it is obvious that $\langle E \rangle \subset \langle F \rangle$ (as subspaces).  Writing $E$ for the columns of $X$ and $F$ for those columns together with $Y$ ($F = E \cup \{\hat Y\}$), we can appreciate the content of the quoted statement: it asserts the inclusion goes the other way, $\langle F \rangle \subset \langle E \rangle$ (from which it follows that these subspaces are equal).
The idea behind the proof is simple: since $\hat Y$ is, by construction, already in $\langle E \rangle$, any linear combination that includes $Y$ can be rewritten as a linear combination involving elements of $E$ alone.
A detailed proof
Rigorous, detailed proofs can be useful for those just learning about vector spaces.  Here is how one might proceed.
Let $f\in \langle F \rangle$.  We intend to show $f\in \langle E \rangle$.  By definition, there exist $f_1, \ldots, f_k\in F$ and scalars $\phi_1, \ldots, \phi_k$ for which 
$$f = \phi_1 f_1 + \phi_2 f_2 + \cdots \phi_k f_k.$$
If none of the $f_i$ is equal to $\hat Y$, this exhibits $f$ as a linear combination of $E$ and we're done.  Otherwise, by changing the order of addition (as allowed by the vector space axioms) we may put all the copies of $\hat Y$ at the end, amounting to $f_l=f_{l+1}=\cdots=f_k=\hat Y$ for some $l$ between $1$ and $k$ inclusive.  Apply the distributive axiom to combine all the coefficients of $\hat Y$, expand $\hat Y = \hat \beta_0 X_0 + \cdots \hat \beta_p X_p$ in terms of elements of $E$, and apply the distributive axiom again.  For brevity, write $\phi_l+\phi_{l+1}+\cdots+\phi_k = \gamma$:
$$\eqalign{
f &= \phi_1 f_1  + \cdots \phi_{l-1} f_{l-1} + (\phi_l+\phi_{l+1}+\cdots+\phi_k)\hat Y \\
&=\phi_1 f_1  + \cdots \phi_{l-1} f_{l-1} + \gamma\left(\hat \beta_0 X_0 + \cdots \hat \beta_p X_p\right) \\
&=\phi_1 f_1  + \cdots \phi_{l-1} f_{l-1} +(\gamma\hat \beta_0) X_0 + \cdots (\gamma\hat \beta_p) X_p .
}$$
This is explicitly a finite linear combination of elements of $E$, because each of the $f_i\in F$ ($i \lt l$) is in $E$ and all the $X_j$ are in $E$, $j=0, 1, \ldots, p$.
Since $f\in \langle F \rangle$ implies $f\in \langle E \rangle$, it follows (by definition) that $\langle F \rangle \subset \langle E \rangle$, QED.
