# Why is it the case that when we try to fit an OLS model to a system with more variables than observations, that the residuals are zero?

I am trying to fit an OLS model to some data, where the number of variables $k$ is greater than the number of observations, $N$. In this case, it is obvious that we will have a infinite amount of solutions for $\hat{\beta} =(X^TX)^{-1}X^TY$. However, I fail to see why the residuals, $Y-X\hat{\beta}$ will be zero for no matter what solution I choose, as long as we are not in the case where there exist NO solutions.

• You are asking why any set $E \subset \mathbb{R}^N$ having more than $N$ vectors must span the entire space. In general, it does not, showing that what is "obvious" is not necessarily true. This suggests that you (as well as the three answers appearing so far) are making some unstated assumptions, which must be tantamount to supposing the variables are of maximal rank. (But if $E$ does span, then--this is the very definition of spanning--any vector $Y$ can be written as a linear combination of $E$, leaving zero residual.) – whuber Jan 27 '15 at 18:02

This is an example of overfitting. With $k \geq N$ you get a model that has a "perfect" fit to the data, so your predictions are the same as values of dependent variable, and so $y - \hat y = 0$.
This does not mean you can just choose any values for $\beta$, have have to choose the right ones that describe a plane that passes through your points. However there will still be an infinite choice of correct solutions.
If you go down to even fewer dimensions, i.e. one point and one independent variable (assuming an offset), that becomes the problem of fitting a straight line through a single point. Let's say the point is at (1,1) we could fit the line with $\beta_0=0$ and $\beta_1=1$, but we could also fit the line with $\beta_0=2$ and $\beta_1=-1$. We have infinite such choices. However we can't just pick any old values. For example $\beta_0=1$ and $\beta_1=1$ won't pass through the point and so the solution is not valid. But for valid solutions, the error is exactly zero, because the line passes exactly through the point. In higher dimensions just think of a hyperplane passing perfectly through every single point, therefore zero residual.