4
$\begingroup$

I'm trying to refresh my memory as to the relationship between ordinary least squares and the linear algebra procedure of projection. (possibly highly related to Linear algebra attributes of a typical data set)

Starting with a model matrix, $\mathbf{X} = [\mathbf{1}, \vec{x_1}, \ldots, \vec{x_p}]$ (with the usual $n$ rows) and a response $Y$, we can define the hat matrix as a projection $\mathbf{H} = \mathbf{X}(\mathbf{X}^t \mathbf{X})^{-1}\mathbf{X}^t $. Then $\hat{Y} = \mathbf{H}Y$. In linear algebra, we define the column space, the row space, and the null space for a matrix.

Considering OLS as a projection:

  • Is the row/column space defined for $\mathbf{X}$? Does the column space correspond to what you can call a "fitted space" or similar, i.e. on $\mathbb{R}^{p}$?
  • Is the "null space" a basis defined by the vector of residuals, i.e. on $\mathbb{R}^{n-p}$?
  • If the model matrix $\mathbf{X}$ contains a duplicate, do we consider the row space to actually be $\mathbb{R}^{n-1}$?
$\endgroup$

1 Answer 1

4
$\begingroup$

Q: Is the row/column space defined for $\mathbf{X}$? Does the column space correspond to what you can call a "fitted space" or similar, i.e. on $\mathbb{R}^{p}$?

A: Defined by $\mathbf{X}$ sounds better to me. I have never heard of a "fitted" space, but "fitting space" sounds like a reasonable term in a regression context. Also, since there are $p+1$ columns, it is a $p+1$-dimensional vector space unless there is a perfect multicollinearity. This vector space is not $\mathbb{R}^{p}$ or $\mathbb{R}^{p+1}$, but rather, it is a subset of $\mathbb{R}^{n}$.

Q: Is the "null space" a basis defined by the vector of residuals, i.e. on $\mathbb{R}^{n-p}$?

A: The residual vector is a point in the (usually) $\{n-(p+1)\}$-dimensional vector space that is the orthogonal complement of the so-called "fitting space." It is not $\mathbb{R}^{n-p}$, but rather it is a subset of $\mathbb{R}^{n}$.

Q: If the model matrix $\mathbf{X}$ contains a duplicate, do we consider the row space to actually be $\mathbb{R}^{n-1}$?

A: Unless there are other collinearities, the column vector space now has dimension $p$ (i.e., $(p+1) -1$). Again, it is not $\mathbb{R}^{n}$ or $\mathbb{R}^{n-1}$; rather, it is a subset of $\mathbb{R}^{n}$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.