Is there an intuitive explanation why multicollinearity is a problem in linear regression?

Question

The wiki discusses the problems that arise when multicollinearity is an issue in linear regression. The basic problem is multicollinearity results in unstable parameter estimates which makes it very difficult to assess the effect of independent variables on dependent variables.

I understand the technical reasons behind the problems (may not be able to invert $X' X$, ill-conditioned $X' X$ etc) but I am searching for a more intuitive (perhaps geometric?) explanation for this issue.

Is there a geometric or perhaps some other form of easily understandable explanation as to why multicollinearity is problematic in the context of linear regression?

Answer 1

Consider the simplest case where $Y$ is regressed against $X$ and $Z$ and where $X$ and $Z$ are highly positively correlated. Then the effect of $X$ on $Y$ is hard to distinguish from the effect of $Z$ on $Y$ because any increase in $X$ tends to be associated with an increase in $Z$.

Another way to look at this is to consider the equation. If we write $Y = b_0 + b_1X + b_2Z + e$, then the coefficient $b_1$ is the increase in $Y$ for every unit increase in $X$ while holding $Z$ constant. But in practice, it is often impossible to hold $Z$ constant and the positive correlation between $X$ and $Z$ mean that a unit increase in $X$ is usually accompanied by some increase in $Z$ at the same time.

A similar but more complicated explanation holds for other forms of multicollinearity.

Answer 2

The geometric approach is to consider the least squares projection of $Y$ onto the subspace spanned by $X$.

Say you have a model:

$E[Y | X] = \beta_{1} X_{1} + \beta_{2} X_{2}$

Our estimation space is the plane determined by the vectors $X_{1}$ and $X_{2}$ and the problem is to find coordinates corresponding to $(\beta_{1}, \beta_{2})$ which will describe the vector $\hat{Y}$, a least squares projection of $Y$ on to that plane.

Now suppose $X_{1} = 2 X_{2}$, i.e. they're colinear. Then, the subspace determined by $X_{1}$ and $X_{2}$ is just a line and we have only one degree of freedom. So we can't determine two values $\beta_{1}$ and $\beta_{2}$ as we were asked.

Answer 3

I was eating sushi once and thought that it might make a good intuitive demonstration of ill-conditioned problems. Suppose you wanted to show someone a plane using two sticks touching at their bases.

You'd probably hold the sticks orthogonal to each other. The effect of any kind of shakiness of your hands on the plane causes it to wobble a little around the what you were hoping to show people, but after watching you for a while they get a good idea of what plane you were intending to demonstrate.

But let's say you bring the sticks' ends closer together and watch the effect of your hands shaking. The plane it forms will pitch far more wildly. Your audience will have to watch longer to get a good idea of what plane the you are trying to demonstrate.

Answer 4

The way I think about this really is in terms of information. Say each of $X_{1}$ and $X_{2}$ has some information about $Y$. The more correlated $X_{1}$ and $X_{2}$ are with each other, the more the information content about $Y$ from $X_{1}$ and $X_{2}$ are similar or overlapping, to the point that for perfectly correlated $X_{1}$ and $X_{2}$, it really is the same information content. If we now put $X_{1}$ and $X_{2}$ in the same (regression) model to explain $Y$, the model tries to "apportion" the information that ($X_{1}$,$X_{2}$) contains about $Y$ to each of $X_{1}$ and $X_{2}$, in a somewhat arbitrary manner. There is no really good way to apportion this, since any split of the information still leads to keeping the total information from ($X_{1}$,$X_{2}$) in the model (for perfectly correlated $X$'s, this really is a case of non-identifiability). This leads to unstable individual estimates for the individual coefficients of $X_{1}$ and $X_{2}$, though if you look at the predicted values $b_{1}X_{1}+b_{2}X_{2}$ over many runs and estimates of $b_{1}$ and $b_{2}$, these will be quite stable.

Answer 5

If two regressors are perfectly correlated, their coefficients will be impossible to calculate; it's helpful to consider why they would be difficult to interpret if we could calculate them. In fact, this explains why it's difficult to interpret variables that are not perfectly correlated but that are also not truly independent.

Suppose that our dependent variable is the daily supply of fish in New York, and our independent variables include one for whether it rains on that day and one for the amount of bait purchased on that day. What we don't realize when we collect our data is that every time it rains, fishermen purchase no bait, and every time it doesn't, they purchase a constant amount of bait. So Bait and Rain are perfectly correlated, and when we run our regression, we can't calculate their coefficients. In reality, Bait and Rain are probably not perfectly correlated, but we wouldn't want to include them both as regressors without somehow cleaning them of their endogeneity.

Answer 6

Assume that two people collaborated and accomplished scientific discovery. It is easy to tell their unique contributions (who did what) when two are totally different persons (one is theory guy and the other is good at experiment), while it is difficult to distinguish their unique influences (coefficients in regression) when they are twins acting similarly.

Answer 7

Two people are pushing a boulder up a hill. You want to know how hard each of them is pushing. Suppose that you watch them push together for ten minutes and the boulder moves 10 feet. Did the first guy do all the work and the second just fake it? Or vice versa? Or 50-50? Since both forces are working at the exact same time, you can't separate the strength of either one separately. All that you can say is that their combined force is 1 foot per minute.

Now imagine that the first guy pushes for a minute himself, then nine minutes with the second guy, and a final minute is just the second guy pushing. Now you can use estimates of forces in the first and last minutes to figure out each person's force separately. Even though they are still largely working at the same time, the fact that there is a bit of difference lets you get estimates of the force for each.

If you saw each man pushing independently for a full ten minutes, that would give you more precise estimates of the forces than if there is a large overlap in the forces.

I leave as an exercise for the reader to extend this case to one man pushing uphill and the other pushing downhill (it still works).

Perfect multicolinearity prevents you from estimating the forces separately; near multicolinearity gives you larger standard errors.

Answer 8

My (very) layman intuition for this is that the OLS model needs a certain level of "signal" in the X variable in order to detect it gives a "good" predicting for Y. If the same "signal" is spread over many X's (because they are correlated), then none of the correlated X's can give enough of a "proof" (statistical significance) that it is a real predictor.

The previous (wonderful) answers do a great work in explaining why that is the case.