An intuitive explanation why correlated predictors will 'inflate' the beta?

Question

I have two predictors in a linear model:

$$y \sim 0.6*x_{1}$$

and

$$y \sim 0.8*x_{2}$$

the bad thing here is that $x_{1}$ and $x_{2}$ are around 60% correlated. when I turn this into a multi-regression, the result becomes:

$$y \sim -0.25* x_{1} + 1.1*x_{2}$$

$\beta_{1}$ changed sign and $\beta_{2}$ becomes bigger.

I cannot get an intuitive view on this change. My guess is that $\beta_{2}$ inflates to capture more variance of $y$, then $\beta_{1}$ becomes negative to 'fine-tune' those 'overshoots'. But I am not sure if my understanding is correct.

Can anyone give an 'intuitive' view on what is happening here?

ps: I have been working with LASSO and ridge so I understand the concept of regularization: I just want to know why the inflation of $\beta$ happens in the first place.

Answer 1

The issue here is multicollinearity. Thinking about the problem geometrically might give some intuition. Because there are two regressors, we're fitting a 2d plane to the data.

First, consider the case where $x_1$ and $x_2$ are perfectly correlated. In this example, I've plotted $y$ without any noise to make it easier to see what's happening, but the situation is the same when $y$ is noisy.

There's a single 'line of best fit' running through the data points, and any plane that intersects this line will produce the same predicted $y$ values (and therefore have the same goodness of fit). The plane can be rotated around this line, giving infinitely many solutions (e.g. the blue and red planes above).

Now suppose that $x_1$ and $x_2$ are highly (but not perfectly) correlated, and $y$ is noisy:

In this case, there's a single plane that fits best (assuming more data points than regressors). But, because $x_1$ and $x_2$ are packed narrowly together, we can wiggle/rotate the plane in certain directions without changing the error much. For example, imagine we rotate the grey plane around a line running through the middle of the point cloud. The left/right edges of the plane will sweep up/down by a large amount, but the values at the data points won't change much.

The solution is therefore unstable, meaning that it's sensitive to the particular data points being fit. If we drew new data points from the same underlying distribution, they would be slightly different. Even if $x_1$ and $x_2$ remain the same, $y$ will be different because of the noise. The original plane might still fit fairly well, but a strongly rotated plane might have slightly smaller error. So, refitting the new data could produce a very different solution.

This also implies that the fit for any particular sample can differ strongly from the true/underlying solution (as you saw in your example). But, notice that what's being affected here is the equation for the plane; the actual predicted $y$ values don't change much in the vicinity of the data points. This means that multicollinearity is a problem for inference (where the goal is to identify the true/underlying coefficents). But, it's not a problem for prediction, as long as predictions are made in the vicinity of the original data.

Decreasing the correlation between the regressors, decreasing the noise in $y$, and/or increasing the number of data points will constrain the solution more strongly.