7
$\begingroup$

I am currently studying Faraway's book "Practical Regression and Anova using R". In the very beginning, he says this:

enter image description here

and then

enter image description here

Now, I understand that $y\in\mathbb{R}^n$ since we would have $n$ obervations, then $\beta\in\mathbb{R}^p$ since we have $p$ variables and hence $p$ weights attached to those variables, but why are the residuals in $\mathbb{R}^{n-p}$? I don't understand, since by my interpretation, the residuals are $$\hat{\epsilon_i}=y_i-\hat{y_i}$$ for $i = 1,...,n$. Hence, they would be in $\mathbb{R}^n$? Am I missing something?

$\endgroup$
4
  • 3
    $\begingroup$ The author didn't say "residuals in $\mathbb{R}^{n - p}$". Instead, he says "residuals lie in an $n - p$ dimensional space". More formally, the vector $(\hat{\varepsilon}_1, \ldots, \hat{\varepsilon}_n)$ belongs to a subspace $U$ of $\mathbb{R}^n$, with $\dim(U) = n - p$. $\endgroup$
    – Zhanxiong
    Mar 27, 2021 at 0:25
  • $\begingroup$ @Zhanxiong and why would the dimension be equal to $n-p$? I have no idea how to intuitively interpret this $\endgroup$
    – user274779
    Mar 27, 2021 at 0:30
  • 3
    $\begingroup$ Rigorously, you need some linear algebra knowledge to completely understand it. Do you know the concepts of "subspace" and "dimension" of it? $\endgroup$
    – Zhanxiong
    Mar 27, 2021 at 0:33
  • $\begingroup$ @Zhanxiong yes, I know these. If you could explain it rigorously that would be nice as well $\endgroup$
    – user274779
    Mar 27, 2021 at 0:47

3 Answers 3

8
$\begingroup$

$\newcommand{\rank}{\mathrm{rank}}$ $\newcommand{\tr}{\mathrm{tr}}$ $\newcommand{\real}{\mathbb{R}}$ $\newcommand{\eps}{\epsilon}$ Write the linear model in the matrix form \begin{align*} y = X\beta + \epsilon \end{align*} where $y \in \mathbb{R}^n, X = \begin{pmatrix} x_1 & \cdots & x_{p}\end{pmatrix} \in \mathbb{R}^{n \times p}, \beta \in \mathbb{R}^p, \epsilon \in \mathbb{R}^n$. By convention, $x_1 \equiv e$, where $e \in \real^n$ is a column vector of all ones, and $\rank(X) = p$.

Recall that $H = X(X'X)^{-1}X'$ is the "hat matrix", and the residual vector can be written as $\hat{\epsilon} = (I_{(n)} - H)y$. First note that $y$ lies in $\mathbb{R}^n$, therefore $\hat{\epsilon}$ lies in the image space of the matrix $I_{(n)} - H$ (you can view the matrix $I_{(n)} - H$ as a linear operator on $\mathbb{R}^n$), say $U$. Linear algebra theory asserts that $\dim(U) = \rank(I_{(n)} - H)$. On the other hand, since $I_{(n)} - H$ is idempotent (i.e., $(I_{(n)} - H)^2 = I_{(n)} - H$), its rank is equal to its trace, i.e., \begin{align*} & \rank(I_{(n)} - H) = \tr(I_{(n)} - H) = \tr(I_{(n)}) - \tr(H) \\ = & n - \tr(X(X'X)^{-1}X') = n - \tr(X'X(X'X)^{-1}) = n - \tr(I_{(p)}) = n - p, \end{align*} where we used properties $\tr(A + B) = \tr(A) + \tr(B)$ and $\tr(AB) = \tr(BA)$ of trace.

There is another more geometric flavor argument to derive $\dim(U)$, as your textbook tries to convey. Denote the space spanned by columns of $X$ by $W$, then it is easy to see that \begin{align*} \real^n = W \oplus W^\bot, \tag{$*$} \end{align*} where $W^\bot$ stands for the orthogonal complement of $W$. For any $y \in \real^n$, the decomposition $$y = Hy + (I_{(n)} - H)y = \hat{y} + \hat{\eps}. $$ and $\hat{\eps}'Xa = y'(I - H)Xa = 0$ for any $a \in \real^p$ show that $\hat{\eps} \in W^\bot$, hence $U = W^\bot$. By $(*)$, it follows that \begin{align*} \dim(U) = \dim(W^\bot) = \dim(\real^n) - \dim(W) = n - p, \end{align*} since obviously $\dim(W) = \rank(X) = p$.

$\endgroup$
6
  • $\begingroup$ Is it fair to summarize this by saying that the span of the residuals is reduced by the number of estimated parameters in the regression operation? So you have decomposed the X-space into a parameter space and an error space on the basis of the Y-hat estimation? And that the y-hat subspace forms the analog of a plane in the Faraway's 3-d diagram and the errors are estimated as their orthogonal distances to that plane? (Feel free to correct any errors.) $\endgroup$
    – DWin
    Mar 27, 2021 at 2:01
  • $\begingroup$ @Dwin: I think you need to be a little more careful because "span of residuals" is vague because it may be interpreted as "span of $\hat{\epsilon}_1, \ldots, \hat{\epsilon}_n$", which is meaningless, it is somewhat intrinsically different from saying span of $x_1, \ldots, x_p$. More precisely, I think you are trying to say span of columns of $I_{(n)} - H$, then it is correct. $\endgroup$
    – Zhanxiong
    Mar 27, 2021 at 2:01
  • $\begingroup$ I was trying to steer the discussion back to a geometric interpretation. I do understand that my linear algebra is probably more decrepit even than the term "rusty" would imply. $\endgroup$
    – DWin
    Mar 27, 2021 at 2:03
  • $\begingroup$ @DWin I think it is fair to say "the $n$-space is decomposed to $X$-space and the residual space". $\endgroup$
    – Zhanxiong
    Mar 27, 2021 at 2:03
  • $\begingroup$ Could you say it would be decomposed to a y-hat subspace of span p and a residual subspace of span n-p? $\endgroup$
    – DWin
    Mar 27, 2021 at 2:05
4
$\begingroup$

Orthogonal component

The space for the residuals is the orthogonal complement of the space spanned by $X$.

This is because the residual $\mathbf{y}-\mathbf{\hat{y}}$ is a vector that is perpendicular to the fit $\mathbf{\hat{y}}$. In the geometric representation it is a perpendicular projection of $y$ into the space spanned by the vectors in $X$ namely $\mathbf{\hat y} = X(X^TX)^{-1}X^T \mathbf{y}$.

This orthogonal component has dimension $n-p$


Intuitive illustration

The image in your book is abstract. It might maybe help to draw it for an actual example.

In the example below, you see an illustration for the fitting of $\mathbf{y} = a + b\mathbf{x}$ with only three points.

illustration for a small sample size

The error is a vector perpendicular to the surface spanned by $x_1$ and $x_2$. For any observation, the error will point in the same direction and can be seen to be a multiple of a line (a 1D space).


Linear algebra

You can describe any observation $\mathbf{y}$ (a sample of size $n$) in the space of potential observations as a sum of any $n$ orthogonal vectors. If $p$ of those form the vector $\mathbf{\hat{y}}$ then the remainder (the error $\mathbf{\epsilon}$) is a sum of the $n-p$ other ones.

$$\mathbf{y} = \overbrace{\underbrace{x_1 + x_2 + \dots + x_p}_{\substack{\mathbf{\hat{y}}\\\text{These $p$ vectors/regressors summed}\\\text{form the vector $\mathbf{\hat{y}}$}}} + \underbrace{ e_1 + e_2 + \dots + e_{n-p}}_{\substack{\mathbf{\epsilon}\\\text{The error $\mathbf{\epsilon}$ is a sum of}\\\text{the remaining $n-p$}}}}^{\text{$n$ orthogonal vectors of which $p$ are the regressors $x_i$}}$$

The $n-p$ stems from splitting the space into two orthogonal subspaces, one with dimension $p$, the other with dimension $n-p$.

$\endgroup$
3
  • $\begingroup$ Bravo. That's the sort of geometric illustration I had in mind when I made my comments to @Zhanxiong's answer. You did it much more elegantly and completely than I could have done, although I usually think of the population mean as forming the origin for vectors of the parameter space. $\endgroup$
    – DWin
    Mar 27, 2021 at 16:32
  • $\begingroup$ @DWin fyi the illustration occured before on this site and is originally from stats.stackexchange.com/a/508366 $\endgroup$ Mar 27, 2021 at 20:51
  • $\begingroup$ Another earlier (but slightly varied) occurance is stats.stackexchange.com/a/494419 here it made sense to have the origin of the vectors not in the population mean but in the point $\beta X$ with $\beta =0$. $\endgroup$ Mar 27, 2021 at 21:02
1
$\begingroup$

I think the constraints on the residuals mean that they can be represented by $n-p$ numbers. Take a trivial case,

$$ y = \bar{y} + \varepsilon $$

Here, there will be n error terms, but they will sum to zero. So, if you tell me $n-1$ error terms, I can work out the other one.

I think there is something analogous for the full model, but I'm not sure how to explain it. I guess, if you told me $n-p$ error terms, as well as the $X$ matrix and $\beta$ vector, I would be able to work out the rest.

$\endgroup$
1
  • $\begingroup$ I think it might be more accurate to say that the residuals can be calculated from n-p orthogonal vectors, i.e. the basis for the residual subspace. I suppose a vector could be considered a "number" but I think it's clearer to keep a distiction between the terms number and vector. $\endgroup$
    – DWin
    Mar 27, 2021 at 2:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.