3
$\begingroup$

I am trying to fit an OLS model to some data, where the number of variables $k$ is greater than the number of observations, $N$. In this case, it is obvious that we will have a infinite amount of solutions for $\hat{\beta} =(X^TX)^{-1}X^TY$. However, I fail to see why the residuals, $Y-X\hat{\beta}$ will be zero for no matter what solution I choose, as long as we are not in the case where there exist NO solutions.

$\endgroup$
  • 1
    $\begingroup$ You are asking why any set $E \subset \mathbb{R}^N$ having more than $N$ vectors must span the entire space. In general, it does not, showing that what is "obvious" is not necessarily true. This suggests that you (as well as the three answers appearing so far) are making some unstated assumptions, which must be tantamount to supposing the variables are of maximal rank. (But if $E$ does span, then--this is the very definition of spanning--any vector $Y$ can be written as a linear combination of $E$, leaving zero residual.) $\endgroup$ – whuber Jan 27 '15 at 18:02
1
$\begingroup$

This is an example of overfitting. With $k \geq N$ you get a model that has a "perfect" fit to the data, so your predictions are the same as values of dependent variable, and so $y - \hat y = 0$.

$\endgroup$
1
$\begingroup$

This is the same as when you have more equations than you have unknowns. Imagine you have only 2 observations and two independent variables. This means you are trying to fit a plane to two points. It doesn't matter how those two points are arranged, there exists some plane (actually infinite different planes) that will pass through both of them, hence 0 error (i.e. zero residuals). This same concepts extends to higher dimensions.

This does not mean you can just choose any values for $\beta$, have have to choose the right ones that describe a plane that passes through your points. However there will still be an infinite choice of correct solutions.

If you go down to even fewer dimensions, i.e. one point and one independent variable (assuming an offset), that becomes the problem of fitting a straight line through a single point. Let's say the point is at (1,1) we could fit the line with $\beta_0=0$ and $\beta_1=1$, but we could also fit the line with $\beta_0=2$ and $\beta_1=-1$. We have infinite such choices. However we can't just pick any old values. For example $\beta_0=1$ and $\beta_1=1$ won't pass through the point and so the solution is not valid. But for valid solutions, the error is exactly zero, because the line passes exactly through the point. In higher dimensions just think of a hyperplane passing perfectly through every single point, therefore zero residual.

$\endgroup$
0
$\begingroup$

When there are equal or more parameters than observations, you have one or more ways to fit the model perfectly. It simply becomes linear equation system.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.