0
$\begingroup$

In linear regression, such as:

$$ Y = \beta_0 + \beta_1X_1 + \beta_2X_2 + \epsilon, $$ we are often told that if $X_1$ and $X_2$ are perfectly collinear, we cannot find a solution (non-unique?) because linear regression needs a 2-d space when we have two variables. However, I am failing to understand geometrically why. It seems that if $X_1$ and $X_2$ are perfectly collinear, then they will consist of just a line. However, isnt a projection onto a line unique in the sense that we are just minimizing the distance from the point of a vector $Y$ onto a line? Am I missing something here?

$\endgroup$
  • 1
    $\begingroup$ The projection on the line is unique, but you cannot apportion it to $X_1$ and $X_2$, which is a problem. So you cannot uniquely solve for the coefficients on $X_1$ and $X_2$ even though you can solve for coefficients on the line. $\endgroup$ – Richard Hardy Oct 3 '17 at 7:32
  • $\begingroup$ How do the $\hat{\beta}$ solutions look geometrically if you cannot solve uniquely for them? How does the infiniteness of the solutions look like? $\endgroup$ – user321627 Oct 3 '17 at 7:50
  • $\begingroup$ They are limited by a linear combination of $\beta_1$ and $\beta_2$ being equal to the projection coefficient on the line. Any linear combination that produces the coefficient on the line works. Thus there are infinitely many of possible combinations that only satisfy one equality restriction. $\endgroup$ – Richard Hardy Oct 3 '17 at 8:10
3
$\begingroup$

Richard's comment:

The projection on the line is unique, but you cannot apportion it to $X_1$ and $X_2$, which is a problem. So you cannot uniquely solve for the coefficients on $X_1$ and $X_2$ even though you can solve for coefficients on the line

From a linear algebra point of view, you can define the unique orthogonal projection of the vector onto the linear span of ($X_1$, $X_2$), but not find a unique way to express it as a linear combination of ($X_1$, $X_2$) since they do not form basis (not linearly independent).

Note that this does not mean linear regression is impossible. It just means that there are many solutions.

It may be a problem for interpretation. For prediction, the problem is that you can't easily extrapolate to data where $(X_1,X_2)$ would not be proportional like in the dataset. Assume $X_1=X_2$ and $Y=2X_1$ (i skip the noise for simplicity) in the training set. Some solutions are:

  • $Y=2X_1$
  • $Y=X_1+X_2$ (balanced)
  • $Y=1002X_1-1000X_2$

The last solution would give weird prediction as soon $X_1$ and $X_2$ are slightly different.

The usual method is to use regularization. $L^2$ regularization will choose the balanced solution and will avoid absurd solutions. Note that the regularized solution is always unique (the matrix is invertible).

$\endgroup$
  • $\begingroup$ 'All these expressions predict the same outcome.' For all $(X_1,X_2)\in\mathbb{R}^2$? $\endgroup$ – user603 Oct 4 '17 at 10:50
  • 1
    $\begingroup$ For all $(X_1,X_2)$ in the training set :-). Good point. $\endgroup$ – Benoit Sanchez Oct 4 '17 at 10:52
1
$\begingroup$

If you draw your observations as points on the x-y plane where $X_1$ is on x-axis and $X_2$ on y-axis, then all points will be on a line. They will perfectly lay on one line. That's what collinearity means: $$X_2=a+bX_1$$

So, geometrically you don't need a plane, you don't need two coordinates. All you need is one variable that represents both variables.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.