In Elements of Statistical Learning, the authors note that Linear Regression can be rewritten from

$$ \hat{Y} = \hat{\beta}_0 + \sum_{j = 1}^p X_j\hat{\beta}_j$$


$$ \hat{Y} = X^T \hat{\beta} $$

by including $\hat{\beta}_0$ in the vector of coefficients and including a column of $1$s in $X$. The author's then go on to note that $(X,\hat{Y})$ form a subspace in the $p+1$ dimensional input-output space.

I can't really see why this is true. I understand the axioms of Vector Spaces and Subspaces, but don't understand the argument the authors are making. Could someone please shed some light?

  • $\begingroup$ $\hat{Y}$ is (by definition) the projection into the column space of $X$. Thus, $\hat{Y}$ lives in the column space of $X$. By definition, the column space of $X$ has $p$ dimensions. If $Y$ were linearly dependent on $X$, then there would be no need to run a linear regression (because you could perfectly predict $Y$ from $X$). So you assume that $Y$ is linearly independent from the column space of $X$. Hence, $(Y,X)$ form a $p+1$-dimensional space, but $(\hat{Y}, X)$ form a $p$-dimensional subspace of this space $(Y,X)$. $\endgroup$ – Jeremias K Jan 25 '17 at 16:09
  • $\begingroup$ @JeremiasK That's great, thanks for the explanation. $\endgroup$ – Demetri Pananos Jan 25 '17 at 16:18
  • $\begingroup$ @Jeremias A few things are missing from that explanation. First, it is both trivial and irrelevant that $(X,Y)$ might generate a vector space, because the model does not contemplate forming linear combinations of $X$'s and $Y$. Second, the column space of $X$ does not necessarily have $p$ dimensions (nor must it have $p+1$ dimensions, even though there are $p+1$ columns): that is precisely one of the subtleties underlying the claim that $(X,\hat Y)$ generates a subspace (without specifying its dimension). $\endgroup$ – whuber Jan 25 '17 at 19:30
  • $\begingroup$ @whuber I am not sure what you mean with the first point, can you elaborate? For the second point, are you are saying that I implicitly assume full rank of $X'X$ to get to the $p+1$ dimensions? I agree with that, but I thought that since the question was about Linear Regression/OLS, full rank is implicitly assumed, isn't it? (Otherwise, $\hat{\beta} = (X'X)^{-1}(X'Y)$ has no solution) $\endgroup$ – Jeremias K Jan 26 '17 at 11:48
  • $\begingroup$ @Jeremias When the rank is full, $\hat\beta$ has many solutions. The first point is that the vector space in question does not include $Y$ among its generators, only the columns of $X$. The whole point of the statement is that $\hat Y$ lies (by construction) within the span of the columns of $X$. $\endgroup$ – whuber Jan 26 '17 at 16:24


The sense in which "$(X,\hat Y)$ form a subspace" is that the $p+1$ columns of $X$ (let's call them $X_0, X_1, \ldots, X_{p}$), together with the vector $\hat Y$, generate a subspace of the space generated by the columns of $X$ alone. This assertion embodies several subtleties:

  1. The dimension of this subspace need not be exactly $p+1$, even though $X$ has $p+1$ columns. It can be as small as $1$ (which occurs when there are no regressors at all or when all regressors are identically zero).

  2. The columns of $X$ and $Y$ itself may be elements of infinite-dimensional vector spaces, such as spaces of functions--which include spaces of random variables. This powerful generalization of linear regression comes to us for free, without any additional concepts or calculations being necessary.

  3. It doesn't matter what the field of scalars is. Thus, this result holds even when you are allowing the $\hat \beta$ to be complex numbers. A comparable result holds--taking care to preserve the order of multiplication in every instance--even for noncommutative fields, such as the Quaternions.

  4. The regressors determine the subspace, but the subspace corresponds to many different choices of regressors. It is only the subspace that matters, because linear regression finds (the unique) projection from $Y$ into this subspace. In solving any linear regression problem, therefore, we should think of the regressors solely in terms of the subspace they generate, rather than as individual vectors. This point of view simplifies discussions of identifiability.

What the assertion means

By definition, whenever $E\subset V$ is a subset of a vector space, the subspace it generates consists of all vectors that can be written as finite linear combinations of elements of $E$:

$$\langle E \rangle = \{\alpha_1 e_1 + \alpha_2 e_2 + \cdots \alpha_k e_k\mid e_1,\ldots,e_k\in E\}.$$

When $E\subset F$ it is obvious that $\langle E \rangle \subset \langle F \rangle$ (as subspaces). Writing $E$ for the columns of $X$ and $F$ for those columns together with $Y$ ($F = E \cup \{\hat Y\}$), we can appreciate the content of the quoted statement: it asserts the inclusion goes the other way, $\langle F \rangle \subset \langle E \rangle$ (from which it follows that these subspaces are equal).

The idea behind the proof is simple: since $\hat Y$ is, by construction, already in $\langle E \rangle$, any linear combination that includes $Y$ can be rewritten as a linear combination involving elements of $E$ alone.

A detailed proof

Rigorous, detailed proofs can be useful for those just learning about vector spaces. Here is how one might proceed.

Let $f\in \langle F \rangle$. We intend to show $f\in \langle E \rangle$. By definition, there exist $f_1, \ldots, f_k\in F$ and scalars $\phi_1, \ldots, \phi_k$ for which

$$f = \phi_1 f_1 + \phi_2 f_2 + \cdots \phi_k f_k.$$

If none of the $f_i$ is equal to $\hat Y$, this exhibits $f$ as a linear combination of $E$ and we're done. Otherwise, by changing the order of addition (as allowed by the vector space axioms) we may put all the copies of $\hat Y$ at the end, amounting to $f_l=f_{l+1}=\cdots=f_k=\hat Y$ for some $l$ between $1$ and $k$ inclusive. Apply the distributive axiom to combine all the coefficients of $\hat Y$, expand $\hat Y = \hat \beta_0 X_0 + \cdots \hat \beta_p X_p$ in terms of elements of $E$, and apply the distributive axiom again. For brevity, write $\phi_l+\phi_{l+1}+\cdots+\phi_k = \gamma$:

$$\eqalign{ f &= \phi_1 f_1 + \cdots \phi_{l-1} f_{l-1} + (\phi_l+\phi_{l+1}+\cdots+\phi_k)\hat Y \\ &=\phi_1 f_1 + \cdots \phi_{l-1} f_{l-1} + \gamma\left(\hat \beta_0 X_0 + \cdots \hat \beta_p X_p\right) \\ &=\phi_1 f_1 + \cdots \phi_{l-1} f_{l-1} +(\gamma\hat \beta_0) X_0 + \cdots (\gamma\hat \beta_p) X_p . }$$

This is explicitly a finite linear combination of elements of $E$, because each of the $f_i\in F$ ($i \lt l$) is in $E$ and all the $X_j$ are in $E$, $j=0, 1, \ldots, p$.

Since $f\in \langle F \rangle$ implies $f\in \langle E \rangle$, it follows (by definition) that $\langle F \rangle \subset \langle E \rangle$, QED.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.