4
$\begingroup$

Let $Y = \beta_1 X_1 + \beta_2 X_2 +... + u$ where $u$ is the error term and $X_i's$ are the regressors.

One of the assumptions state that

(1) There is no perfect multicollinearity.

I couldn't really get what would the existence of a perfect multicollinearity do to our coefficients? I understood what perfect multicollinearity means, and all that, but what does it do our regression that somehow it is assumed at the beginning not to exist?

Also, When having perfect multicollinearity, why does dropping the intercept help us avoid it? What I mean is that when two regressors are in a linear relationship for some reason when you set the intercept of the regression $ = 0 $ for some reasons now you avoid multicollinearity?

$\endgroup$
  • $\begingroup$ I answered your first question below, but I wasn't sure what you mean by the last question. If you edit, I'll try to answer. $\endgroup$ – Dave Kielpinski Oct 11 '17 at 0:18
  • $\begingroup$ I think the second question is around: if you encode a categorical feature fully, then combined with the intercept you have perfect collinearity. $\endgroup$ – Matthew Drury Oct 11 '17 at 0:35
6
$\begingroup$

I bet this was covered a million times on this board. In a nutshell: because the design matrix becomes degenerated, and there is no unique solution to the linear algebra problem of OLS. There will be infinite number of equally good solutions, and there's no way to tell which one is better.

Technical details: the design matrix is a matrix that is constructed by putting all $p$ variables in columns and all $n$ observations in rows. It is $X_{ij}$ where $i$ is rows from 1 to $n$ and $j$ is columns from 1 to $p$. It so happens that when there is a perfect collinearity then matrix $X$ can be reduced to a matrix $X'_{ik}$ where each column now represents a new set of variables $k=[1, p']$ such that $p'<p$. In other words the new design matrix $X'$ has fewer columns than the original, yet no information was lost.

In this case the usual solution $\beta=(X^TX)^{-1}X^TY$ does not exist because $(X^TX)^{-1}$ is singular. On the other hand the solution $\beta'=(X'^TX')^{-1}X'^TY$ does exist on the new set of variables. So, the only problem with perfect collinearity is that the original set of variables does not have a unique solution, but it does have solutions.

The implication is that you can pick any of the non unique solutions, and it will be as good as any other. Note, it will not be as bad as any other. So, you can use this solution to predict $Y'$. The only problem's that you'll have to step outside a typical OLS method to find the solutions, because OLS' linear algebra trick doesn't work. Things like gradient descent will work.

$\endgroup$
  • $\begingroup$ Oh finally something that makes sense! It is hard to learn statistics as an undergraduate since they skip all the technical details. Eyvallah. $\endgroup$ – Xenidia Oct 11 '17 at 0:35
  • $\begingroup$ When you're learning stats skip what you don't understand and focus on what's fun to play with. You can't swallow the whole thing during one meal, you'll get back to these things when time is right. $\endgroup$ – Aksakal Oct 11 '17 at 0:37
7
$\begingroup$

Perfect multicollinearity leads to great pain. Suppose your data $Y$ is generated by a single parameter $X$ with an added noise process $u$, so $Y = \beta X + u$.

Now let me (foolishly!) adopt a model $Y = \beta_1 X_1 + \beta_2 X_2 + u$, where there is perfect multicollinearity, say $X_1 = X_2$. I try to find $\beta_1, \beta_2$ by regression. But the least-squares error is minimized equally well for many solutions, just as long as $\beta_1 + \beta_2 = \beta$. So in fact, there's no way to state independent values for $\beta_1$ and $\beta_2$ in this case. There is also no way to state confidence intervals for $\beta_1$ and $\beta_2$. Personally, I don't regard any statistical quantity as meaningful unless I can give a confidence interval.

$\endgroup$
  • 1
    $\begingroup$ I think saying "regression is meaningless" is a bit too much. You just have non-unique, equally valid, solutions. $\endgroup$ – Matthew Drury Oct 11 '17 at 0:23
  • 1
    $\begingroup$ @MatthewDrury edited to address your point $\endgroup$ – Dave Kielpinski Oct 11 '17 at 0:26
  • $\begingroup$ I'd say interpretation of each individual beta is meaningless because it can be anything, while interpretation of collective effect of all betas can be useful. $\endgroup$ – Akavall Oct 11 '17 at 1:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.