Why is it the case that when we try to fit an OLS model to a system with more variables than observations, that the residuals are zero? I am trying to fit an OLS model to some data, where the number of variables $k$ is greater than the number of observations, $N$. In this case, it is obvious that we will have a infinite amount of solutions for $\hat{\beta} =(X^TX)^{-1}X^TY$. However, I fail to see why the residuals, $Y-X\hat{\beta}$ will be zero for no matter what solution I choose, as long as we are not in the case where there exist NO solutions. 
 A: This is the same as when you have more equations than you have unknowns. Imagine you have only 2 observations and two independent variables. This means you are trying to fit a plane to two points. It doesn't matter how those two points are arranged, there exists some plane (actually infinite different planes) that will pass through both of them, hence 0 error (i.e. zero residuals). This same concepts extends to higher dimensions.
This does not mean you can just choose any values for $\beta$, have have to choose the right ones that describe a plane that passes through your points. However there will still be an infinite choice of correct solutions.
If you go down to even fewer dimensions, i.e. one point and one independent variable (assuming an offset), that becomes the problem of fitting a straight line through a single point. Let's say the point is at (1,1) we could fit the line with $\beta_0=0$ and $\beta_1=1$, but we could also fit the line with $\beta_0=2$ and $\beta_1=-1$. We have infinite such choices. However we can't just pick any old values. For example $\beta_0=1$ and $\beta_1=1$ won't pass through the point and so the solution is not valid. But for valid solutions, the error is exactly zero, because the line passes exactly through the point. In higher dimensions just think of a hyperplane passing perfectly through every single point, therefore zero residual.
A: This is an example of overfitting. With $k \geq N$ you get a model that has a "perfect" fit to the data, so your predictions are the same as values of dependent variable, and so $y - \hat y = 0$.
A: When there are equal or more parameters than observations, you have one or more ways to fit the model perfectly. It simply becomes linear equation system.
