# Minimum numbers of observations for OLS (MLE) depending on number of parameters

Say I want to minimize my error term $\epsilon_i = y_i - ( \sum_{n=1}^{N} \beta_n x_{in} )$ given N observed inputs $x_{in}$ and one output $y_i$ for I observations.

I am looking for the optimal parameters $\beta$ that best describe my observations. Let's just assume the $\epsilon$ are normal distributed, so we can do a OLS estimation $\min \sum_{i=1}^I \epsilon_i^2$.

I heard there is the requirement, that $I > N+1$. But I fail to understand why.

This is because it will lead to your system being under-determined and therefore will have infinitely many solutions.

For example consider the case where $N=2$ and $I=1$. This means you are trying to fit a straight line through a single point. There are an infinite number of straight lines that will fit this point with precisely $0$ error.

The same happen if $N=3$ and $I<3$ as here you are fitting a 2-D plane to either a single point (clearly infinite solutions) or to two points. Two point make a line and you can rotate the plane around the orthogonal direction to that line thus again giving you infinitely many solutions.

Think of $N$ as your degrees of freedom and $I$ as your constraints. If you have fewer constraints than degrees of freedom then you will have an infinite number of solutions.

By contrast when $I>N$ the system is likely to be over-determined and therefore there is no solution with $0$ error which is why we used OLS to find a best fit.

Here is a poorly made illustration:

Top left image shows $N=2$ and $I=1$, you can see that the green, red and blue lines will all have zero error but also have completely different values for $\beta$ where as top right shows $N=2$ and $I=2$. Now there is only one solution (error is still zero). Bottom left shows the same thing as the top left for $N=3$, $I=2$ and bottom right attempts to show that when $N=3$ and $I=3$ there is only one 2-D plane that can pass through all three points.

• thx, that was helpful. Jan 15 '18 at 11:24