# What is the significance of estimation space and error space in linear models?

Consider a linear model $$E(y)=X\beta,\,\operatorname{Var}(y)=\sigma^2I_n$$ where $$y$$ is an $$n\times 1$$ response vector, $$\beta$$ is a $$p\times 1$$ vector of parameters and $$X$$ is an $$n\times p$$ design matrix with $$\operatorname{rank}(X)=r\le p\,(.

In the book Plane Answers to Complex Questions by Ronald Christensen, there is this concept of estimation space and error space in relation to a linear model. Estimation space is defined as the column space of $$X$$ and error space is defined as the orthogonal complement of the estimation space, i.e. the null space of $$X^T$$. I am trying to understand the significance of this definition. Does the estimation space represent the vector space of all estimable linear functions of $$\beta$$? What does the error space represent? Can I say that it is the vector space of all functions $$C^Ty$$ such that $$E(C^Ty)=0$$? I can see that dimension of estimation space is $$r$$ and dimension of error space is $$n-r$$.

The estimation space of your design matrix $$X$$, let's call this space $$E_X$$, is exactly the linear subspace of response vectors $$y\in\mathbb{R}^n$$ that you can "reach" with a model from this design matrix. I.e. it is the set of all $$y$$ for which there is a $$\beta$$ such that $$y=X\beta$$. Each point of $$E_X$$ belongs to a unique $$\beta$$.
Note that all your n measurements are combined into this one response vector $$y$$, so we are only talking about a single point in $$\mathbb{R}^n$$.
Usually, $$n > \dim(\beta)$$, so $$E_X$$ is a proper subspace. Your measured response vector $$y$$ is usually positioned outside of $$E_X$$, and one approach of estimating $$\beta$$ is to find the point $$p\in E_X$$ that is nearest to $$y$$ and then return the unique $$\beta$$ of that point $$p$$. And the method of ordinary least squares (OLS) does exactly that: it finds the orthogonal projection ($$P_{E_X}$$) of $$y$$ onto $$E_X$$: $$y_{OLS} = P_{E_X}(y).$$ Note, that, and this is generally true, orthogonal projection of any point $$q$$ to any subspace $$W$$ is giving you the point in $$W$$ that is nearest to $$q$$ in the Euclidean metric $$\|p-q\|_2$$: $$P_W(q)\in W,\quad\mbox{and}\quad\|P_W(q) - q\|_2 = min_{p\in W} \|q - p\|_2.$$ Then, the difference between the measured $$y$$ and the OLS estimate, i.e. the projection onto $$E_X$$, is the error: $$e_X(y) = y - P_{E_X}(y).$$ Since OLS is an orthogonal projection, $$e_X(y)$$ is orthogonal to $$E_X$$ and thus an element of the error space. Hence the name.
I'd like to add to frank's answer. Estimation space is the column space of $$X$$, denoted $$\mathcal{C}(X)$$. The space of estimable linear functions of $$\beta$$ is, in a sense, the row space of $$X$$, i.e., $$\mathcal{C}(X^T)$$. They are not the same. By the definition of estimability, we have that $$c'\beta\ \text{is estimable}\iff c\in\mathcal{C}(X^T).$$ In addition, there are only $$r=\mathrm{rank}(X)$$ linearly independent estimable linear functions.
Just like orthogonally decomposing a vector in $$\mathbb{R}^n$$ onto the estimation space $$\mathcal{C}(X)$$ and the error space $$\mathcal{N}(X^T)$$, we can decompose a vector in $$\mathbb{R}^p$$ onto $$\mathcal{C}(X^T)$$ and $$\mathcal{N}(X)$$. Suppose $$c'\beta$$ is estimable. Then $$c\in\mathcal{C}(X^T)$$. For any $$u=u_1+u_2$$ with $$u_1\in\mathcal{C}(X^T)$$ and $$u_2\in\mathcal{N}(X)$$, we can see that $$c'u=c'(u_1+u_2)=c'u_1$$ because every vector in $$\mathcal{C}(X^T)$$ is orthogonal to every vector in $$\mathcal{N}(X)$$. Geometrically speaking, to find a solution to the linear system $$X^TX\beta=X^Ty$$, we can first project $$y$$ onto $$\mathcal{C}(X)$$ to get $$\hat{y}$$, which is equal to $$Xu$$ for some $$u\in\mathbb{R}^p$$, and then find a specific solution $$u_1\in\mathcal{C}(X^T)$$ such that $$Xu_1=\hat{y}$$. Such $$u_1$$ is unique, because $$\mathcal{C}(X^T)$$ and $$\mathcal{C}(X)$$ has the same dimension and hence are $$1$$-to-$$1$$ corresponded. Hence, a solution for $$\beta$$ is $$\beta^0=u_1+u_2$$, for any $$u_2\in\mathcal{N}(X)$$. It follows that $$c'\beta^0=c'u_1$$ is unique.