# Showing that multiple linear regression breaks down with less variables than parameters to estimate

I consider a multiple linear regression model with only two explanatory variables.

$$\ y=\beta_0+\beta_1x_1+\beta_2x_2+\epsilon$$

fulfilling the CLM properties.

From the moment restrictions

$$\frac{1}{n}\sum_{i=1}^n(y_i+\hat{\beta_0}+\hat{\beta_1}x_{i1}+\hat{\beta_2}x_{i2})=0$$

$$\frac{1}{n}\sum_{i=1}^nx_{i1}(y_i+\hat{\beta_0}+\hat{\beta_1}x_{i1}+\hat{\beta_2}x_{i2})=0$$

$$\frac{1}{n}\sum_{i=1}^nx_{i2}(y_i+\hat{\beta_0}+\hat{\beta_1}x_{i1}+\hat{\beta_2}x_{i2})=0$$

I want to show in a simple way that the model breaks down without using matrix algebra if I only have two observations ($$n=2$$).

I have managed to do it with a simple linear regression model (only one independent variable) with only one observation ($$n=1$$), where the moment restrictions are:

$$\frac{1}{n}\sum_{i=1}^n(y_i+\hat{\beta_0}+\hat{\beta_1}x_{i})=0$$

$$\frac{1}{n}\sum_{i=1}^nx_{i}(y_i+\hat{\beta_0}+\hat{\beta_1}x_{i})=0$$

Since we have $$n=1$$ it follows that the moment restrictions can also be written as:

$$y_{1}-\hat{\beta}_{0}-\hat{\beta}_{1}x_{1}=0$$

$$x_{1}\left(y_{1}-\hat{\beta}_{0}-\hat{\beta}_{1}x_{1}\right)=0$$

Now Isolating for $$\hat{\beta_{0}}$$ in the first equation:

$$\hat{\beta_{0}}=y_{1}-\hat{\beta}_{1}x_{1}$$

And plugging this into the second equation I get:

$$x_{1}\left(y_{1}-\left(y_{1}-\hat{\beta}_{1}x_{1}\right)-\hat{\beta}_{1}x_{1}\right)=0\Leftrightarrow x_1\cdot0=0$$

Showing that things break down.

But I have really problems with doing it in a simple way in the multiple linear regression framework.

Hope that someone can help.

• Is there any good reason why you are hampering yourself by not using matrix algebra? You're essentially asking to reproduce a basic theorem of linear algebra (about dimensions of subspaces) while demanding that we not use the tools or concepts of linear algebra. The exercise seems pointless. – whuber Feb 10 at 13:37
• Not other than my linear algebra is not that good. But if you have a solution using linear algebra or alternatively can refer me to the basic theorem, it would be a great help. – Rasmus12 Feb 10 at 18:15
• The basic theorem asserts the dimension of the image plus the dimension of the kernel of a linear map of vector spaces sum to the dimension of its domain. Translation: there is more than one solution to any set of $n$ simultaneous linear equations in $p$ variables when $n \lt p.$ – whuber Feb 10 at 22:24
• Possibly an even more direct translation: Consider $n=2$. With $p=2$ (typically, a constant and a regressor) you can put a line through two points. Now suppose $p=3$. In that case, you fit a plane, and there are infinitely many different planes that share these two points. – Christoph Hanck Feb 11 at 12:15