Linear regression as a projection

I'm trying to refresh my memory as to the relationship between ordinary least squares and the linear algebra procedure of projection. (possibly highly related to Linear algebra attributes of a typical data set)

Starting with a model matrix, $$\mathbf{X} = [\mathbf{1}, \vec{x_1}, \ldots, \vec{x_p}]$$ (with the usual $$n$$ rows) and a response $$Y$$, we can define the hat matrix as a projection $$\mathbf{H} = \mathbf{X}(\mathbf{X}^t \mathbf{X})^{-1}\mathbf{X}^t$$. Then $$\hat{Y} = \mathbf{H}Y$$. In linear algebra, we define the column space, the row space, and the null space for a matrix.

Considering OLS as a projection:

• Is the row/column space defined for $$\mathbf{X}$$? Does the column space correspond to what you can call a "fitted space" or similar, i.e. on $$\mathbb{R}^{p}$$?
• Is the "null space" a basis defined by the vector of residuals, i.e. on $$\mathbb{R}^{n-p}$$?
• If the model matrix $$\mathbf{X}$$ contains a duplicate, do we consider the row space to actually be $$\mathbb{R}^{n-1}$$?

1 Answer

Q: Is the row/column space defined for $$\mathbf{X}$$? Does the column space correspond to what you can call a "fitted space" or similar, i.e. on $$\mathbb{R}^{p}$$?

A: Defined by $$\mathbf{X}$$ sounds better to me. I have never heard of a "fitted" space, but "fitting space" sounds like a reasonable term in a regression context. Also, since there are $$p+1$$ columns, it is a $$p+1$$-dimensional vector space unless there is a perfect multicollinearity. This vector space is not $$\mathbb{R}^{p}$$ or $$\mathbb{R}^{p+1}$$, but rather, it is a subset of $$\mathbb{R}^{n}$$.

Q: Is the "null space" a basis defined by the vector of residuals, i.e. on $$\mathbb{R}^{n-p}$$?

A: The residual vector is a point in the (usually) $$\{n-(p+1)\}$$-dimensional vector space that is the orthogonal complement of the so-called "fitting space." It is not $$\mathbb{R}^{n-p}$$, but rather it is a subset of $$\mathbb{R}^{n}$$.

Q: If the model matrix $$\mathbf{X}$$ contains a duplicate, do we consider the row space to actually be $$\mathbb{R}^{n-1}$$?

A: Unless there are other collinearities, the column vector space now has dimension $$p$$ (i.e., $$(p+1) -1$$). Again, it is not $$\mathbb{R}^{n}$$ or $$\mathbb{R}^{n-1}$$; rather, it is a subset of $$\mathbb{R}^{n}$$.