3
$\begingroup$

I have a model y ~ x + z and the correlation between x and z is 0.2. This is only weakly positive. So, how seriously should I consider the effects of multicollinearity? Is it worth expressing a simpler model like y ~ x?

$\endgroup$
  • 6
    $\begingroup$ Correlation of 0.2 is really nothing to worry about. $\endgroup$ – Richard Hardy Feb 12 '15 at 15:10
13
$\begingroup$

As mentioned, 0.2 is nothing to worry about in your specific case. But to answer the question, "how seriously should I consider the effect of multicollinearity in my regression model?" recognize that multicollinearity does not reduce the ability of the model to be predictive (you may still do a good job predicting your response variable in the presence of multicollinearity). It does, however, affect the quality of the interpretation of the individual predictors. In other words, in the presence of multicollinearity you may not have reliable results (indicated by large standard errors) about any individual $\beta$ coefficient.

This is intuitive if we keep in mind what the interpretation of a $\beta_i$ coefficient is. It is the effect of a 1-unit increase in it's corresponding independent variable, $X_i$, holding all other variables constant. When $X_i$ is highly correlated with some of the other independent variables then it tends to change when the other variables change. In other words, the data doesnt tend to show us what happens when $X_i$ changes and all other variables are constant. Consequently, it becomes more difficult to say the effect of $X_i$ holding all other variables constant.

$\endgroup$
  • $\begingroup$ -1, I don't really agree w/ this. I don't know how to interpret the claim "multicollinearity does not reduce the ability of the model to be predictive", nor the claim "in the presence of multicollinearity you may not have valid results about any individual β coefficient". I do think "it does... affect interpretation of the individual predictors" is wrong; the interpretation is identical. The fact that Xi is highly correlated does not mean that "it changes when the other variables change", only that higher values of Xi tend to be associated w/ higher (lower) values of Xj. $\endgroup$ – gung Feb 12 '15 at 15:59
  • $\begingroup$ What if I said, "it does affect the quality of the interpretation of the individual coefficients". The interpretation of the $\beta$ stays the same it's just that the standard errors will go up in the presence of multicollinearity reducing the quality of interpreting the point estimate. $\endgroup$ – TrynnaDoStat Feb 12 '15 at 16:07
  • $\begingroup$ The SEs do increase, but what "the quality of the interpretation" is is rather opaque. There are several issues here (although if you can correct them, I am happy to remove my downvote). $\endgroup$ – gung Feb 12 '15 at 16:12
  • $\begingroup$ Please let me know if you are satisfied. I kept the sentence "In other words, we don't observe situations in the data when $X_i$ changes and all other variables are constant" because the opposite of constant is change and I wanted to make this clear. Of course, not all change is a linear relationship but the OP seems to have an understanding of the precise definition of correlation. $\endgroup$ – TrynnaDoStat Feb 12 '15 at 16:25
  • $\begingroup$ Unless there is perfect multicollinearity, there is no reason why you cannot have a change in Xi w/ all other variables held constant. There is nothing necessary or causal about the relationship amongst the variables when they are correlated. You may be confusing multicollinearity w/, eg, an interaction term. $\endgroup$ – gung Feb 12 '15 at 16:47
7
$\begingroup$

If you have perfect multicollinearity (i.e., $r_{xz}=1$), then there is no unique set of numbers (i.e., $\hat\beta_x$ and $\hat\beta_z$) that will optimize the fit of the data. Moreover, the typical method software uses to fit a model will fail and you may simply get an error message in this case. People sometimes define multicollinearity as only perfect multicollinearity, and this restrictive definition lies behind claims that multicollinearity renders the preferred model invalid.

On the other hand, most people do not restrict multicollinearity to perfect linear dependence. If the multicollinearity isn't perfect (which in your case simply means the correlation isn't perfect, since you have only two variables), then there will be a unique solution and the model can be fit. However, if you were to draw a new sample and re-run your study identically, you would find that the beta estimates vary more widely than you might otherwise suspect. The sampling distributions of your betas remain unbiased (meaning that they are centered on the true value), but the width (i.e., the standard error) expands. In simple cases like yours, it is easy to calculate how much the variance expands. Specifically, you can calculate the variance inflation factor:
$$ {\rm VIF} = \frac{1}{1-r_{xz}^2} $$ In addition, if $x$ and $z$ are correlated, the bivariate sampling distribution of $\hat\beta_x$ and $\hat\beta_z$ will have non-zero covariance. Furthermore, the more strongly $x$ and $z$ are correlated, the larger the covariance. This point is less well-known, however, and typically is not what people are primarily concerned about.

In your specific case, the variances of the sampling deviations of your betas (both of them) will be $1.04$ times as wide (i.e., the standard errors will be $1.02$ times as wide) as they would have been if $r_{xz} = 0.\bar0$ and everything else about your study (e.g., $N$) were the same. It strikes me as very unlikely that you would need to worry about this (although strictly speaking, I don't know). So in answer to your explicit question, I would not worry about multicollinearity in your case, and would not try a simpler model.


This is just about all there is to say about multicollinearity—the alpha and the psy, as it were; it is what should be said about multicollinearity in the abstract / without knowing a lot of details about a particular study. For the sake of clarity, let me explicitly state some additional things, lest they be left ambiguous by dint of omission.

Because of the existence of multicollinearity, your model will / will not have sufficient predictive utility.

This cannot be said without more information.

Your coefficients will not be sufficiently reliable.

This cannot be said without more information.

Your variables will be non-significant.

This cannot be said without more information.

The interpretation of the coefficients that would normally hold in a multiple regression model does not apply, or is invalid or problematic, when there is multicollinearity.

The interpretation of a coefficient in a multiple regression model is the change in $Y$ associated with a one-unit change in $X_i$ holding all else equal (there is more information here, if needed). This can remain true even when there are substantial correlations amongst the $X$ variables. Consider that, in the following plot, $x_1$ and $x_2$ are correlated $r_{x_1x_2} = .99$ and yet there are many cases where $x_1$ differs in value but $x_2$ is constant:

enter image description here

To understand this more clearly, it may help to contrast it with some well-known cases where the standard interpretation above is invalid or problematic. Consider the coefficient for $X_i$ when there is an interaction term including $X_i$, say $X_iX_j$. In that case, there is no such thing as "a one-unit change in $X_i$ holding all else equal", as it cannot be guaranteed that the interaction term can be held equal. Hence, the coefficient for $X_i$ turns out to be the change in $Y$ associated with a one-unit change in $X_i$ when $X_j=0$ (and all other variables are held equal). Similarly, if there is an $X_i^2$ term in the model, it cannot be held equal when $X_i$ changes. Although I have argued that it is best to interpret all polynomial terms involving the variable together, the coefficient on $X_i$ is related to the location in $X_i$ of the global extremum of the parabola. The point of these contrasting examples is that multicollinearity is not necessarily like this. As the figure above shows, there is nothing about the existence of multicollinearity that necessitates it being impossible for $X_i$ to vary while $X_j$ is constant, and so the standard interpretation of the coefficients remains generally valid / tenable. (Note, however, that interaction terms and squared terms typically induce multicollinearity, so there can be situations with multicollinearity where the normal interpretation does not apply.)

To further clarify these points, I simulated the model $Y = 2 + .03X_1 - .03X_2 + \varepsilon$, with $\varepsilon\sim\mathcal N(0, 1)$ using $N = 225$ vs. $22500$, and using the above pattern of $x_1$ and $x_2$ vs. having them sampled on a uniform from $[0,\ 50]$. Plots of the bivariate sampling distributions of the estimated coefficients are below:

enter image description here

Despite a ${\rm VIF}$ of $85.3$, with high $N$, the power of both $\hat\beta_1$ and $\hat\beta_2$ approach $1.0$.

$\endgroup$
  • $\begingroup$ With perfect multicollinearity the model can be fit--but the fit is not unique. $\endgroup$ – whuber Feb 12 '15 at 17:29
  • $\begingroup$ @whuber, I suppose I'm conflating two claims there. 1) the design matrix is not invertable, so software will return an error instead of a fit (but there are ways around this), & 2) the model is unidentifiable / there is no unique solution for the parameter estimates. I can clarify this, if you think it best. I left this simpler for this OP. $\endgroup$ – gung Feb 12 '15 at 17:35
  • $\begingroup$ I figured that's what you intended, Gung. Note, though, that most software will not return an error: it will attempt to resolve the issue by eliminating variables, and it usually succeeds. Identifiability is worth discussing, especially if a distinction is made based on the purpose of the regression: it's not as much of an issue when the regression is performed solely for prediction. I think @Trynna did a good job covering these issues in another answer in this thread. $\endgroup$ – whuber Feb 12 '15 at 17:40
  • $\begingroup$ That's interesting, @whuber, I have several problems w/ Trynna's post, as I tried to point out in comments & it currently being discussed in chat. $\endgroup$ – gung Feb 12 '15 at 17:48
  • 1
    $\begingroup$ I took a look and--although I upvoted that post in the belief it offered clear, concise, helpful advice--I do agree with the points you are making in chat. The "tends" language and the apparently causal analysis in terms of "if you change this, that will change" could deceive naive readers. I am aware of two effective explanations that avoid these solecisms: the geometric one (of OLS as an orthogonal projection) and the one offered by Tukey & Mosteller, who use sequential "matching" to describe multiple regression and explain how near-collinearity inflates the SEs of the coefficient estimates. $\endgroup$ – whuber Feb 12 '15 at 17:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.