I was doing some simulations in R where I have a predictor matrix $X \in \mathbb{R}^{n\times p}$ and a response vector $y \in \mathbb{R}^n$. I fitted the linear model with the lm command and compute the sum of square residuals $|y_i-X\hat{\beta}_i|^2$. Now, if this is done with $p>n$ then the sum of square residuals is always $0$, why is this so?
3
-
$\begingroup$ @AdamO edited, thanks $\endgroup$– EjrionmCommented Sep 27, 2021 at 17:54
-
2$\begingroup$ This question is predicated on a false assumption. Simple counterexample: $X=\pmatrix{1&1&1\\1&1&1}$ and $y=\pmatrix{0\\1}$ has nonzero residuals, yet $3=p\gt n=2.$ $\endgroup$– whuber ♦Commented Sep 27, 2021 at 18:12
-
2$\begingroup$ This onslaught of basic OLS questions in past days on CV can mean only one thing: another MOOC has just started. $\endgroup$– AksakalCommented Sep 27, 2021 at 19:02
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
6
If $p > n$ then you have more parameters than observations and your model can fit the data perfectly, which is equivalent to saying that the residuals are zero.
Note that although your model is a perfect fit, it isn't very helpful as it's completely overfitted to your data and is unlikely to generalise well to out-of-sample data.
-
$\begingroup$ Is $p$ the dimension of $X$ or the rank? $\endgroup$– AdamOCommented Sep 27, 2021 at 16:49
-
$\begingroup$ @AdamO I was thinking about the number of parameters being estimated here - so it'd be the dimension of $X$. Please let me know if that's not correct. $\endgroup$ Commented Sep 27, 2021 at 16:51
-
1$\begingroup$ You actually need the rank of $X$ to be bigger than $n$ to guarantee perfect prediction. Otherwise, the span of $X$ may be less is required to express each $Y$ exactly as a linear combination of the rows of $X$. $\endgroup$– AdamOCommented Sep 27, 2021 at 16:56
-
$\begingroup$ @AdamO Makes perfect sense, thanks. $\endgroup$ Commented Sep 27, 2021 at 17:04
-
$\begingroup$ @AdriàLuz thanks, I understand the reasoning, but I'd like to have some math intuition, so you're saying that in the case of $p>n$ we have $X \hat{\beta}_i = y_i$? $\endgroup$– EjrionmCommented Sep 27, 2021 at 18:12
$\begingroup$
$\endgroup$
1
Some say the LS solution is not solvable when $p > n$ but that's not actually true, it's not unique. If the design matrix $X$ is full rank then it can predict every observation in the $Y$ vector perfectly, and the projection into the null space is null. The least squares solution:
$$ \hat{\beta} = \left( X^T X \right)^{-1} X^T Y$$
requires that $\left( X^T X \right)^{-1}$ is calculated using a pseudoinverse. Whereas an inverse has $X^T X = I$ a pseudo inverse has $X X^{-1} X = X$.
-
$\begingroup$ "the projection into the null space is null" can you please add more math details on that? I'd like to see the math behind that behavior on the sse $\endgroup$– EjrionmCommented Sep 27, 2021 at 17:59