Say I want to minimize my error term $\epsilon_i = y_i - ( \sum_{n=1}^{N} \beta_n x_{in} )$ given N observed inputs $x_{in}$ and one output $y_i$ for I observations.

I am looking for the optimal parameters $\beta$ that best describe my observations. Let's just assume the $\epsilon$ are normal distributed, so we can do a OLS estimation $\min \sum_{i=1}^I \epsilon_i^2$.

I heard there is the requirement, that $I > N+1$. But I fail to understand why.


This is because it will lead to your system being under-determined and therefore will have infinitely many solutions.

For example consider the case where $N=2$ and $I=1$. This means you are trying to fit a straight line through a single point. There are an infinite number of straight lines that will fit this point with precisely $0$ error.

The same happen if $N=3$ and $I<3$ as here you are fitting a 2-D plane to either a single point (clearly infinite solutions) or to two points. Two point make a line and you can rotate the plane around the orthogonal direction to that line thus again giving you infinitely many solutions.

Think of $N$ as your degrees of freedom and $I$ as your constraints. If you have fewer constraints than degrees of freedom then you will have an infinite number of solutions.

By contrast when $I>N$ the system is likely to be over-determined and therefore there is no solution with $0$ error which is why we used OLS to find a best fit.

Here is a poorly made illustration:

enter image description here

Top left image shows $N=2$ and $I=1$, you can see that the green, red and blue lines will all have zero error but also have completely different values for $\beta$ where as top right shows $N=2$ and $I=2$. Now there is only one solution (error is still zero). Bottom left shows the same thing as the top left for $N=3$, $I=2$ and bottom right attempts to show that when $N=3$ and $I=3$ there is only one 2-D plane that can pass through all three points.

  • 1
    $\begingroup$ thx, that was helpful. $\endgroup$
    – Xraaz
    Jan 15 '18 at 11:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.