What is the significance of estimation space and error space in linear models?

Question

Consider a linear model $E(y)=X\beta,\,\operatorname{Var}(y)=\sigma^2I_n$ where $y$ is an $n\times 1$ response vector, $\beta$ is a $p\times 1$ vector of parameters and $X$ is an $n\times p$ design matrix with $\operatorname{rank}(X)=r\le p\,(<n)$.

In the book Plane Answers to Complex Questions by Ronald Christensen, there is this concept of estimation space and error space in relation to a linear model. Estimation space is defined as the column space of $X$ and error space is defined as the orthogonal complement of the estimation space, i.e. the null space of $X^T$. I am trying to understand the significance of this definition. Does the estimation space represent the vector space of all estimable linear functions of $\beta$? What does the error space represent? Can I say that it is the vector space of all functions $C^Ty$ such that $E(C^Ty)=0$? I can see that dimension of estimation space is $r$ and dimension of error space is $n-r$.

Answer 1

The estimation space of your design matrix $X$, let's call this space $E_X$, is exactly the linear subspace of response vectors $y\in\mathbb{R}^n$ that you can "reach" with a model from this design matrix. I.e. it is the set of all $y$ for which there is a $\beta$ such that $y=X\beta$. Each point of $E_X$ belongs to a unique $\beta$.

Note that all your n measurements are combined into this one response vector $y$, so we are only talking about a single point in $\mathbb{R}^n$.

Usually, $n > \dim(\beta)$, so $E_X$ is a proper subspace. Your measured response vector $y$ is usually positioned outside of $E_X$, and one approach of estimating $\beta$ is to find the point $p\in E_X$ that is nearest to $y$ and then return the unique $\beta$ of that point $p$. And the method of ordinary least squares (OLS) does exactly that: it finds the orthogonal projection ($P_{E_X}$) of $y$ onto $E_X$: $$y_{OLS} = P_{E_X}(y).$$ Note, that, and this is generally true, orthogonal projection of any point $q$ to any subspace $W$ is giving you the point in $W$ that is nearest to $q$ in the Euclidean metric $\|p-q\|_2$: $$ P_W(q)\in W,\quad\mbox{and}\quad\|P_W(q) - q\|_2 = min_{p\in W} \|q - p\|_2. $$ Then, the difference between the measured $y$ and the OLS estimate, i.e. the projection onto $E_X$, is the error: $$ e_X(y) = y - P_{E_X}(y). $$ Since OLS is an orthogonal projection, $e_X(y)$ is orthogonal to $E_X$ and thus an element of the error space. Hence the name.

With OLS, everything is nice beautiful linear geometry. But as soon as you introduce e.g. regularization, the geometry gets much more complicated.

Answer 2

I'd like to add to frank's answer. Estimation space is the column space of $X$, denoted $\mathcal{C}(X)$. The space of estimable linear functions of $\beta$ is, in a sense, the row space of $X$, i.e., $\mathcal{C}(X^T)$. They are not the same. By the definition of estimability, we have that $$c'\beta\ \text{is estimable}\iff c\in\mathcal{C}(X^T).$$ In addition, there are only $r=\mathrm{rank}(X)$ linearly independent estimable linear functions.

Just like orthogonally decomposing a vector in $\mathbb{R}^n$ onto the estimation space $\mathcal{C}(X)$ and the error space $\mathcal{N}(X^T)$, we can decompose a vector in $\mathbb{R}^p$ onto $\mathcal{C}(X^T)$ and $\mathcal{N}(X)$. Suppose $c'\beta$ is estimable. Then $c\in\mathcal{C}(X^T)$. For any $u=u_1+u_2$ with $u_1\in\mathcal{C}(X^T)$ and $u_2\in\mathcal{N}(X)$, we can see that $$c'u=c'(u_1+u_2)=c'u_1$$ because every vector in $\mathcal{C}(X^T)$ is orthogonal to every vector in $\mathcal{N}(X)$. Geometrically speaking, to find a solution to the linear system $X^TX\beta=X^Ty$, we can first project $y$ onto $\mathcal{C}(X)$ to get $\hat{y}$, which is equal to $Xu$ for some $u\in\mathbb{R}^p$, and then find a specific solution $u_1\in\mathcal{C}(X^T)$ such that $Xu_1=\hat{y}$. Such $u_1$ is unique, because $\mathcal{C}(X^T)$ and $\mathcal{C}(X)$ has the same dimension and hence are $1$-to-$1$ corresponded. Hence, a solution for $\beta$ is $\beta^0=u_1+u_2$, for any $u_2\in\mathcal{N}(X)$. It follows that $c'\beta^0=c'u_1$ is unique.