# Least squares residuals

I was doing some simulations in R where I have a predictor matrix $$X \in \mathbb{R}^{n\times p}$$ and a response vector $$y \in \mathbb{R}^n$$. I fitted the linear model with the lm command and compute the sum of square residuals $$|y_i-X\hat{\beta}_i|^2$$. Now, if this is done with $$p>n$$ then the sum of square residuals is always $$0$$, why is this so?

• @AdamO edited, thanks Sep 27 '21 at 17:54
• This question is predicated on a false assumption. Simple counterexample: $X=\pmatrix{1&1&1\\1&1&1}$ and $y=\pmatrix{0\\1}$ has nonzero residuals, yet $3=p\gt n=2.$
– whuber
Sep 27 '21 at 18:12
• This onslaught of basic OLS questions in past days on CV can mean only one thing: another MOOC has just started. Sep 27 '21 at 19:02

If $$p > n$$ then you have more parameters than observations and your model can fit the data perfectly, which is equivalent to saying that the residuals are zero.

Note that although your model is a perfect fit, it isn't very helpful as it's completely overfitted to your data and is unlikely to generalise well to out-of-sample data.

• Is $p$ the dimension of $X$ or the rank? Sep 27 '21 at 16:49
• @AdamO I was thinking about the number of parameters being estimated here - so it'd be the dimension of $X$. Please let me know if that's not correct. Sep 27 '21 at 16:51
• You actually need the rank of $X$ to be bigger than $n$ to guarantee perfect prediction. Otherwise, the span of $X$ may be less is required to express each $Y$ exactly as a linear combination of the rows of $X$. Sep 27 '21 at 16:56
• @AdamO Makes perfect sense, thanks. Sep 27 '21 at 17:04
• @AdriàLuz thanks, I understand the reasoning, but I'd like to have some math intuition, so you're saying that in the case of $p>n$ we have $X \hat{\beta}_i = y_i$? Sep 27 '21 at 18:12

Some say the LS solution is not solvable when $$p > n$$ but that's not actually true, it's not unique. If the design matrix $$X$$ is full rank then it can predict every observation in the $$Y$$ vector perfectly, and the projection into the null space is null. The least squares solution:

$$\hat{\beta} = \left( X^T X \right)^{-1} X^T Y$$

requires that $$\left( X^T X \right)^{-1}$$ is calculated using a pseudoinverse. Whereas an inverse has $$X^T X = I$$ a pseudo inverse has $$X X^{-1} X = X$$.

• "the projection into the null space is null" can you please add more math details on that? I'd like to see the math behind that behavior on the sse Sep 27 '21 at 17:59