I have two predictors in a linear model:

$$y \sim 0.6*x_{1}$$


$$y \sim 0.8*x_{2}$$

the bad thing here is that $x_{1}$ and $x_{2}$ are around 60% correlated. when I turn this into a multi-regression, the result becomes:

$$y \sim -0.25* x_{1} + 1.1*x_{2}$$

$\beta_{1}$ changed sign and $\beta_{2}$ becomes bigger.

I cannot get an intuitive view on this change. My guess is that $\beta_{2}$ inflates to capture more variance of $y$, then $\beta_{1}$ becomes negative to 'fine-tune' those 'overshoots'. But I am not sure if my understanding is correct.

Can anyone give an 'intuitive' view on what is happening here?

ps: I have been working with LASSO and ridge so I understand the concept of regularization: I just want to know why the inflation of $\beta$ happens in the first place.


The issue here is multicollinearity. Thinking about the problem geometrically might give some intuition. Because there are two regressors, we're fitting a 2d plane to the data.

First, consider the case where $x_1$ and $x_2$ are perfectly correlated. In this example, I've plotted $y$ without any noise to make it easier to see what's happening, but the situation is the same when $y$ is noisy.

enter image description here

There's a single 'line of best fit' running through the data points, and any plane that intersects this line will produce the same predicted $y$ values (and therefore have the same goodness of fit). The plane can be rotated around this line, giving infinitely many solutions (e.g. the blue and red planes above).

Now suppose that $x_1$ and $x_2$ are highly (but not perfectly) correlated, and $y$ is noisy:

enter image description here

In this case, there's a single plane that fits best (assuming more data points than regressors). But, because $x_1$ and $x_2$ are packed narrowly together, we can wiggle/rotate the plane in certain directions without changing the error much. For example, imagine we rotate the grey plane around a line running through the middle of the point cloud. The left/right edges of the plane will sweep up/down by a large amount, but the values at the data points won't change much.

The solution is therefore unstable, meaning that it's sensitive to the particular data points being fit. If we drew new data points from the same underlying distribution, they would be slightly different. Even if $x_1$ and $x_2$ remain the same, $y$ will be different because of the noise. The original plane might still fit fairly well, but a strongly rotated plane might have slightly smaller error. So, refitting the new data could produce a very different solution.

This also implies that the fit for any particular sample can differ strongly from the true/underlying solution (as you saw in your example). But, notice that what's being affected here is the equation for the plane; the actual predicted $y$ values don't change much in the vicinity of the data points. This means that multicollinearity is a problem for inference (where the goal is to identify the true/underlying coefficents). But, it's not a problem for prediction, as long as predictions are made in the vicinity of the original data.

Decreasing the correlation between the regressors, decreasing the noise in $y$, and/or increasing the number of data points will constrain the solution more strongly.

  • 1
    $\begingroup$ I had not realized there were a distinction between inference and prediction. Here is a related question stats.stackexchange.com/questions/244017/… . The difference is somewhat nuanced: apparently inference refers to the structures/models that describe a process whereas prediction focuses on the outcomes. In your example the randomization and noise could lead to significantly different coefficients (and thus distinct inference) but the outcomes/results are still relatively similar (prediction). $\endgroup$ – StephenBoesch Jan 8 '18 at 0:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.