1
$\begingroup$

Many sources (here is one of many https://towardsdatascience.com/assumptions-of-linear-regression-algorithm-ed9ea32224e1) state that linear regression assumes there is little to no multicollinearity between the independent variables. I don't understand why this is the case.

Linear regression assumes the model looks like the following: $$ Y = X\beta + \epsilon $$

There is nothing about this model that assumes little to no multicollinearity. Now if you were to approximate this model with least squares, then yes, if there were to be perfect multicollinearity, then your least squares estimators are no longer unique, or if there were strong multicollinearity, but not perfect, then the condition number of $(X^TX$ will be large, meaning that $\hat{\beta}$ is unstable, but even then there is no such assumption about "little or no multicollinearity."

So why do sources claim that linear regression assumes multicollinearity?

In my opinion, the only thing that linear regression assumes is that $Y = X\beta + \epsilon$, i.e., $Y$ varies linearly with $X$ subject to some unobservable $\epsilon$. That's it. No other assumptions. A simple Google search for "linear regression assumptions" will produce numerous links that the assumptions of linear regression as "normal distribution of errors," "little to no multicollinearity," "Homoscedasticity," among others. I just don't understand how these are "assumptions." They're more like implicit requirements for the underlying data in order for linear regression to produce a good fit.

It might be correct to say that in order to evaluate a certain linear regression method (e.g., least squares), we should assume x, y, z, but saying that linear regression assumes x, y, z seems misleading.

Here's another link that stood out to me https://medium.com/@dhiraj8899/top-5-assumptions-for-linear-regression-dd8d10bb0039. In the beginning, it states "For a linear regression algorithm to work properly, it has to pass at least the following five assumptions." I don't know if this is poor wording, or if I'm being anal, or what, but assumptions aren't what's going to make an algorithm work properly. Making an assumption has nothing to do with your algorithm working.

$\endgroup$
5
  • $\begingroup$ Perfect multicollinearity can be an issue sometimes, not just any $\endgroup$
    – Aksakal
    Commented May 22, 2020 at 4:32
  • 1
    $\begingroup$ The premise is incorrect. It’s not linear regression assumption. I wouldn’t count that web site as a reliable source on stats. They can say whatever they want but multicollinearity is often benign issue if that $\endgroup$
    – Aksakal
    Commented May 22, 2020 at 4:39
  • 1
    $\begingroup$ The last article on medium is laughable. Quit reading this garbage. “All variables multivariate normal” , I’ll puke if I finish reading this $\endgroup$
    – Aksakal
    Commented May 22, 2020 at 4:42
  • $\begingroup$ @Aksakal Yeah, unfortunately, these sort of articles are what appears at the top when I search for "linear regression assumptions," which is kind of scary. $\endgroup$
    – 24n8
    Commented May 22, 2020 at 5:00
  • $\begingroup$ this blog makes fun of the outsized attention given to multicolinearity. If your data is colinear, that maybe just reflects the fact that the word often is multicolinear! $\endgroup$ Commented May 22, 2020 at 7:25

2 Answers 2

1
$\begingroup$

There is nothing about the model that assumes anything about co-linearity. The first link is misleading (and the second is just wrong).

Co-linearity is an issue in interpretation, though. Further down in that first linked document it says

The stronger the correlation, the more difficult it is to change one feature without changing another. It becomes difficult for the model to estimate the relationship between each feature and the target independently because the features tend to change in unison.

That's still not quite true, but it's pointing in an important direction. Suppose you want a model where the coefficient of $X$ describes the effect of changing $X$ either (less ambitiously) on your prediction or (more ambitiously) on the actual outcome variable $Y$.

Before we understood causal inference, the simplest way to think about this was that a designed experiment gave you uncorrelated predictors, and the coefficients were effects of changing those predictors, and if you had observational data where the predictors were basically uncorrelated, the coefficients would still have those interpretations.

This wasn't quite right, because it was thinking in terms of correlations, which aren't the right causal quantities (eg they are symmetric, which cause and effect aren't), but it was a helpful way to think if you didn't have better tools. And it was true to say that interpreting the coefficients is hard if the correlations are not small.

We have better tools now: if you want to set up a model where a coefficient has a causal interpretation, we know the criteria it will have to satisfy (and it probably won't satisfy them).

Also, generalisability to other distributions of $X$ can tend to be worse with co-linearity: if the 'training' data do not vary much in one direction, and your new data are different along that one direction, it would be optimistic to expect the model to generalise well. So, if you have co-linearity you'd want to know whether it was a property just of your particular data set or of the whole phenomenon you're studying.

In summary: no, it isn't an assumption of the model, but it is (or at least used to be) an assumption of the people doing the modelling, for not entirely bad reasons.

$\endgroup$
0
$\begingroup$

Forgive me, but I must say that "source" is not someone who speaks of something, and "reliable source" is not what appears at the top in a Google search.

You should better look at Arthur Goldberger, A Course in Econometrics. The assumptions of "classical" regression are:

  1. $E(y)=X\beta$
  2. $V(y)=\sigma^2 I$
  3. $X$ non stochastic
  4. $X$ full rank

The assumptions of "neoclassical" regression (with $X$ random) are:

  1. $E(y|X)=X\beta$
  2. $V(y|X)=\sigma^2 I$
  3. $X$ stochastic
  4. $X$ full rank

Period. As to multicollinearity there is a famous chaper in that book, summarised by Bruce Hansen, Econometrics, pages 124-126: it was just a concern overemphasized by some earlier textbooks.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.