106
$\begingroup$

The wiki discusses the problems that arise when multicollinearity is an issue in linear regression. The basic problem is multicollinearity results in unstable parameter estimates which makes it very difficult to assess the effect of independent variables on dependent variables.

I understand the technical reasons behind the problems (may not be able to invert $X' X$, ill-conditioned $X' X$ etc) but I am searching for a more intuitive (perhaps geometric?) explanation for this issue.

Is there a geometric or perhaps some other form of easily understandable explanation as to why multicollinearity is problematic in the context of linear regression?

$\endgroup$
2
  • 4
    $\begingroup$ Really great question. The best way to understand something is from multiple direction of explanation. $\endgroup$
    – Tal Galili
    Aug 3, 2010 at 2:29
  • 1
    $\begingroup$ See also related question and visual explanation stats.stackexchange.com/q/70899/3277 $\endgroup$
    – ttnphns
    Dec 19, 2013 at 19:26

9 Answers 9

115
$\begingroup$

Consider the simplest case where $Y$ is regressed against $X$ and $Z$ and where $X$ and $Z$ are highly positively correlated. Then the effect of $X$ on $Y$ is hard to distinguish from the effect of $Z$ on $Y$ because any increase in $X$ tends to be associated with an increase in $Z$.

Another way to look at this is to consider the equation. If we write $Y = b_0 + b_1X + b_2Z + e$, then the coefficient $b_1$ is the increase in $Y$ for every unit increase in $X$ while holding $Z$ constant. But in practice, it is often impossible to hold $Z$ constant and the positive correlation between $X$ and $Z$ mean that a unit increase in $X$ is usually accompanied by some increase in $Z$ at the same time.

A similar but more complicated explanation holds for other forms of multicollinearity.

$\endgroup$
2
  • 30
    $\begingroup$ +1 The extremely pathological case where $X = Z$ highlights this further. $Y = b_0 + b_1 X + b_2 Z + e$ and $Y = b_0 + (b_1 + b_2) X + 0 Z + e$ would be indistinguishable. $\endgroup$
    – vqv
    Dec 20, 2010 at 6:41
  • 1
    $\begingroup$ +1 I like this answer because one of the most common help questions is why then is $b_1 > 0$ and $b_2 < 0$. Inference has to account for realistic inputs. $\endgroup$
    – muratoa
    Aug 20, 2012 at 18:52
36
$\begingroup$

I was eating sushi once and thought that it might make a good intuitive demonstration of ill-conditioned problems. Suppose you wanted to show someone a plane using two sticks touching at their bases.

You'd probably hold the sticks orthogonal to each other. The effect of any kind of shakiness of your hands on the plane causes it to wobble a little around the what you were hoping to show people, but after watching you for a while they get a good idea of what plane you were intending to demonstrate.

But let's say you bring the sticks' ends closer together and watch the effect of your hands shaking. The plane it forms will pitch far more wildly. Your audience will have to watch longer to get a good idea of what plane the you are trying to demonstrate.

$\endgroup$
4
  • $\begingroup$ +1 I think this most directly answers the question. Because although multicollinearity affects interpretation. Why it is an issue imho is the stability in estimation. $\endgroup$
    – muratoa
    Aug 20, 2012 at 18:54
  • $\begingroup$ +1 For posting this comment (and only this comment ever in the history of Stackoverflow) under the username Snackrifice. $\endgroup$ May 16, 2018 at 12:19
  • $\begingroup$ I've read this comment probably ten times over several years of stumbling upon it and still not sure what you're saying. What kind of "plane" are you talking about? What do you mean "[the] plane you were intending to demonstrate?" $\endgroup$ Feb 13, 2020 at 15:42
  • $\begingroup$ @Parseltongue The plane this answer is trying to demonstrate is the plane made by the estimation of the target value by a linear combination of the features, a.k.a. the end result of linear regression with 2 features. This analogy demonstrates that a high correlation between features results in a high variance in your model. Also, when the number of features is not equal to 2, the "plane" is actually a line/hyperplane. $\endgroup$
    – Noah May
    Jan 2, 2021 at 12:15
23
$\begingroup$

The geometric approach is to consider the least squares projection of $Y$ onto the subspace spanned by $X$.

Say you have a model:

$E[Y | X] = \beta_{1} X_{1} + \beta_{2} X_{2}$

Our estimation space is the plane determined by the vectors $X_{1}$ and $X_{2}$ and the problem is to find coordinates corresponding to $(\beta_{1}, \beta_{2})$ which will describe the vector $\hat{Y}$, a least squares projection of $Y$ on to that plane.

Now suppose $X_{1} = 2 X_{2}$, i.e. they're collinear. Then, the subspace determined by $X_{1}$ and $X_{2}$ is just a line and we have only one degree of freedom. So we can't determine two values $\beta_{1}$ and $\beta_{2}$ as we were asked.

$\endgroup$
3
  • 4
    $\begingroup$ I upvoted a long ago but re-reading your answer it remind me that I always liked Plane Answers to Complex Questions from Christensen (j.mp/atRp9w). $\endgroup$
    – chl
    Oct 11, 2010 at 21:14
  • $\begingroup$ @chl: cool, definitely going to check it out then. :) $\endgroup$
    – ars
    Oct 12, 2010 at 2:30
  • 1
    $\begingroup$ any answer that begins with "consider the least squares projection of 𝑌 onto the subspace spanned by 𝑋" is not intuitive. $\endgroup$ Feb 13, 2020 at 15:44
17
$\begingroup$

Two people are pushing a boulder up a hill. You want to know how hard each of them is pushing. Suppose that you watch them push together for ten minutes and the boulder moves 10 feet. Did the first guy do all the work and the second just fake it? Or vice versa? Or 50-50? Since both forces are working at the exact same time, you can't separate the strength of either one separately. All that you can say is that their combined force is 1 foot per minute.

Now imagine that the first guy pushes for a minute himself, then nine minutes with the second guy, and a final minute is just the second guy pushing. Now you can use estimates of forces in the first and last minutes to figure out each person's force separately. Even though they are still largely working at the same time, the fact that there is a bit of difference lets you get estimates of the force for each.

If you saw each man pushing independently for a full ten minutes, that would give you more precise estimates of the forces than if there is a large overlap in the forces.

I leave as an exercise for the reader to extend this case to one man pushing uphill and the other pushing downhill (it still works).

Perfect multicolinearity prevents you from estimating the forces separately; near multicolinearity gives you larger standard errors.

$\endgroup$
7
$\begingroup$

The way I think about this really is in terms of information. Say each of $X_{1}$ and $X_{2}$ has some information about $Y$. The more correlated $X_{1}$ and $X_{2}$ are with each other, the more the information content about $Y$ from $X_{1}$ and $X_{2}$ are similar or overlapping, to the point that for perfectly correlated $X_{1}$ and $X_{2}$, it really is the same information content. If we now put $X_{1}$ and $X_{2}$ in the same (regression) model to explain $Y$, the model tries to "apportion" the information that ($X_{1}$,$X_{2}$) contains about $Y$ to each of $X_{1}$ and $X_{2}$, in a somewhat arbitrary manner. There is no really good way to apportion this, since any split of the information still leads to keeping the total information from ($X_{1}$,$X_{2}$) in the model (for perfectly correlated $X$'s, this really is a case of non-identifiability). This leads to unstable individual estimates for the individual coefficients of $X_{1}$ and $X_{2}$, though if you look at the predicted values $b_{1}X_{1}+b_{2}X_{2}$ over many runs and estimates of $b_{1}$ and $b_{2}$, these will be quite stable.

$\endgroup$
4
$\begingroup$

My (very) layman intuition for this is that the OLS model needs a certain level of "signal" in the X variable in order to detect it gives a "good" predicting for Y. If the same "signal" is spread over many X's (because they are correlated), then none of the correlated X's can give enough of a "proof" (statistical significance) that it is a real predictor.

The previous (wonderful) answers do a great work in explaining why that is the case.

$\endgroup$
3
$\begingroup$

Assume that two people collaborated and accomplished scientific discovery. It is easy to tell their unique contributions (who did what) when two are totally different persons (one is theory guy and the other is good at experiment), while it is difficult to distinguish their unique influences (coefficients in regression) when they are twins acting similarly.

$\endgroup$
2
$\begingroup$

If two regressors are perfectly correlated, their coefficients will be impossible to calculate; it's helpful to consider why they would be difficult to interpret if we could calculate them. In fact, this explains why it's difficult to interpret variables that are not perfectly correlated but that are also not truly independent.

Suppose that our dependent variable is the daily supply of fish in New York, and our independent variables include one for whether it rains on that day and one for the amount of bait purchased on that day. What we don't realize when we collect our data is that every time it rains, fishermen purchase no bait, and every time it doesn't, they purchase a constant amount of bait. So Bait and Rain are perfectly correlated, and when we run our regression, we can't calculate their coefficients. In reality, Bait and Rain are probably not perfectly correlated, but we wouldn't want to include them both as regressors without somehow cleaning them of their endogeneity.

$\endgroup$
1
$\begingroup$

I think the dummy variable trap provides another useful possibility to illustrate why multicollinearity is a problem. Recall that it arises when we have a constant and a full set of dummies in the model. Then, the sum of the dummies adds up to one, the constant, so multicollinearity.

E.g., a dummy for men and one for women:

$$y_i=\beta_0+\beta_1Man_i+\beta_2Woman_i+u_i$$

The standard interpretation of $\beta_1$ is the expected change in $Y$ that arises from changing $Man_i$ from 0 to 1. Likewise, $\beta_2$ is the expected change in $Y$ that arises from changing $Woman_i$ from 0 to 1.

But, what is $\beta_0$ then supposed to represent? It is $E(y_i|Man_i=0,Woman_i=0)$, so the expected outcome for persons who are neither a man nor a woman. So if everybody in the dataset identifies as either man or woman, $\beta_0$ represents nobody.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.