0
$\begingroup$

I was doing some simulations in R where I have a predictor matrix $X \in \mathbb{R}^{n\times p}$ and a response vector $y \in \mathbb{R}^n$. I fitted the linear model with the lm command and compute the sum of square residuals $|y_i-X\hat{\beta}_i|^2$. Now, if this is done with $p>n$ then the sum of square residuals is always $0$, why is this so?

$\endgroup$
3
  • $\begingroup$ @AdamO edited, thanks $\endgroup$
    – Ejrionm
    Sep 27 at 17:54
  • 2
    $\begingroup$ This question is predicated on a false assumption. Simple counterexample: $X=\pmatrix{1&1&1\\1&1&1}$ and $y=\pmatrix{0\\1}$ has nonzero residuals, yet $3=p\gt n=2.$ $\endgroup$
    – whuber
    Sep 27 at 18:12
  • 2
    $\begingroup$ This onslaught of basic OLS questions in past days on CV can mean only one thing: another MOOC has just started. $\endgroup$
    – Aksakal
    Sep 27 at 19:02
2
$\begingroup$

If $p > n$ then you have more parameters than observations and your model can fit the data perfectly, which is equivalent to saying that the residuals are zero.

Note that although your model is a perfect fit, it isn't very helpful as it's completely overfitted to your data and is unlikely to generalise well to out-of-sample data.

$\endgroup$
6
  • $\begingroup$ Is $p$ the dimension of $X$ or the rank? $\endgroup$
    – AdamO
    Sep 27 at 16:49
  • $\begingroup$ @AdamO I was thinking about the number of parameters being estimated here - so it'd be the dimension of $X$. Please let me know if that's not correct. $\endgroup$
    – Adrià Luz
    Sep 27 at 16:51
  • 1
    $\begingroup$ You actually need the rank of $X$ to be bigger than $n$ to guarantee perfect prediction. Otherwise, the span of $X$ may be less is required to express each $Y$ exactly as a linear combination of the rows of $X$. $\endgroup$
    – AdamO
    Sep 27 at 16:56
  • $\begingroup$ @AdamO Makes perfect sense, thanks. $\endgroup$
    – Adrià Luz
    Sep 27 at 17:04
  • $\begingroup$ @AdriàLuz thanks, I understand the reasoning, but I'd like to have some math intuition, so you're saying that in the case of $p>n$ we have $X \hat{\beta}_i = y_i$? $\endgroup$
    – Ejrionm
    Sep 27 at 18:12
2
$\begingroup$

Some say the LS solution is not solvable when $p > n$ but that's not actually true, it's not unique. If the design matrix $X$ is full rank then it can predict every observation in the $Y$ vector perfectly, and the projection into the null space is null. The least squares solution:

$$ \hat{\beta} = \left( X^T X \right)^{-1} X^T Y$$

requires that $\left( X^T X \right)^{-1}$ is calculated using a pseudoinverse. Whereas an inverse has $X^T X = I$ a pseudo inverse has $X X^{-1} X = X$.

$\endgroup$
1
  • $\begingroup$ "the projection into the null space is null" can you please add more math details on that? I'd like to see the math behind that behavior on the sse $\endgroup$
    – Ejrionm
    Sep 27 at 17:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.