in these days I was thinking about the collinearity or multicollinearity problem in a linear regression model. Let's suppose we have a model like this:


When we build the model we need to be worried about the predictors, in this case:$X_1,X_2,X_3$ and we hope that they're not a linear combination among them.

So, for example: $X_1=a+bX_2+cX_3$

The problem comes when there's almost multicollinearity (and not exactly). The reason behind is clear to me. If this happens we would have a redundancy about the information of $X_1$.
So what we're asking, actually, is that predictors should be indipendent among them.

But what if there's not a linear, but a quadratic (or others) relationship between them. Suppose $X_2=X_1^2$. In this case they're not indipendent.

But when I build a linear regression model I'm just worried only about linearity.
Shouldn't I be worried even about other kinds of relationships among predictors? Maybe the answer has to do with the fact that, in a multivariate gaussian distribution, incorrelation implies indipendency. But this answer is not convincing me at all.

So why do we concern only on linearity?


1 Answer 1


There are really two types of linearity you may be concerned with when building a linear regression model to solve a problem

  1. Linearity between the predictors and the response: $y = \beta_0 + \beta_1 x_1 + \cdots + \beta_k x_k + \epsilon$.

  2. Linearity amongst the predictors themselves: $x_j \approx a + b x_2 + c x_3 $.

In your question you've called out the second as of concern, but understanding the first is arguably more important.

The first linearity concern is about fitting the data well (more accurately, fitting the process that generated the data well). This is needed if you want anything you learn from the model to accurately reflect what is really going on in the universe. This type of linearity is in no way guaranteed. If you collect some $x$ and $y$ data, throw them in a regression, and call it a day, you can't count on this type of relationship being true, and hence you can not count on anything you learn from such a model generalizing to the true state of the world.

For example, if I collect some $x$ and $y$ data, and the true relationship is like this

$$ y_{true} = a + b x + c x^2 + \epsilon $$

But I get lazy and fit this model

$$ y_{model} = a + b x + \epsilon $$

Then anything I claim to learn from the fit model is almost surely nonsense (unless the effect of $x^2$ is very small, and hence ignorable, but I wouldn't know that...).

This is why people spend so much effort on model specification, collecting many predictors, using polynomials and splines, etc. We need the model to fit the data well for our knowledge to generalize.

The second type of linearity comes into play when we want to derive a specific type of knowledge from the fit model. If we want to interpret the fit coefficients as telling us something about the relationship between two quantities in the universe, then we better hope that our model has accurately estimated the true relationship. As you mention, the more closely related two predictors in the model are, the more difficulty the model will have discriminating the effects of predictor one from predictor two. Close to linear effects amongst the predictors make precise parameter estimation difficult.

The reason that a linear relationship is problematic for precise estimation comes down to the mathematics of linear regression. Recall (this is proven in any good exposition on the mathematics of regression inference) that the covariance of the estimated parameters in regression is given by:

$$ cov(\hat \beta, \hat \beta) = \sigma (X^t X)^{-1} $$

In this answer I discuss the two varaible case, which is representative. Under the assumption that we have centered and standardized our predictors (which we are always free to do, this is just changing our units of measurement), the solution for the first parameter estimate is

$$ \hat \beta_1 = \frac{cov(X_1, y) - cov(X_1, X_2) cov(X_2, y)}{(1 - cov(X_1, X_2)^2) } $$

Notice the denominator $1 - cov(X_1, X_2)^2$. When $X_1$ and $X_2$ are close to colinear, the covariance (which under our centering and standardization is the correlation) is close to one, so the denominator is close to zero. This means that any errors we make in precisely estimating the true correlation between $x_1$ and $x_2$ will be blown up (as it will be a small error in estimating a close to zero denominator, so the reciprocal is a huge error). This is what causes the difficulty in precisely estimating $\hat \beta_1$.

There is clearly a tension between the first and second concern. The more we want our model to be flexible, so that it can adapt to the true complexities in the things we are studying, the more difficulty the model will have precisely estimating the effects we are after. This is fundamental, a kind of uncertainty principle all users of data must learn to balance. This is why there is so much effort in modern day poured into treating large scale inferences: regularization, bayesian method, model validation, they help us find balance between these two competing concerns.

  • 3
    $\begingroup$ This is all a good explanation, but it seems to me that it misses the main point of OP's question, i.e. whether (and why) only linear relationships between predictors are problematic: "But what if there's not a linear, but a quadratic (or others) relationship between them... In this case they're not independent." $\endgroup$ Nov 5, 2017 at 19:48
  • $\begingroup$ @RoseHartman Ok. I added some more discussion about the point you raise. Please check it out. $\endgroup$ Nov 5, 2017 at 20:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.