# Coefficient of highly correlated variables under LASSO and ridge

I have been presented with some interesting questions but unfortunately, I am struggling to provide satisfactory answers. The questions are as follows:

How will the regression coefficients of two variables be impacted if their correlation is 0.999 in OLS? What are the implications for Lasso and Ridge? And how do we determine which variable to keep in practical terms?

I have attempted to address the first question by stating that the coefficients will become highly unstable and sensitive to even minor changes in the response variable. However, I am unsure if there exists any quantitative relationship between the coefficients of these two variables that can be further elaborated upon.

As for the implications of Lasso and Ridge, and the criteria for choosing which variable to retain, I am currently at a loss. I am seeking guidance and insight. Any assistance would be greatly appreciated.

Thank you.

A geometric representation of lasso and ridge is often made by a plot of the iso-lines of the penalty and the squared residuals as function of the parameters. The estimate will be in a place where these lines touch.

If the regressors correlate a lot then you get, in this plot, a shape for this squared residuals that is a very elongated ellipse.

The lasso estimate has the tendency to give a sparse solution and depending on the data it can be either a large $$\hat\beta_1$$ or $$\hat\beta_2$$ (the instability is strongest when the ellipse is at a 45° angle, and it wiggles around that if the regressors $$X$$ are variable, but this does not need to be, if the variance of $$\hat\beta_1$$ and $$\hat\beta_2$$ differs a lot or when $$X$$ is fixed).

The ridge estimate will be giving a solution that is more a blend of the two variables.

In short, ridge has a tendency to give an estimate that is closer to the line $$\beta_1=\beta_2$$. Lasso has a tendency to give an estimate that is closer to the line $$\beta_1=0$$ or $$\beta_2=0$$.

Note that in the image the shape of the ellipse is defined by the regressors $$X$$ and the position/center is defined by the outcome $$y$$. So when we change $$y$$ while keeping $$X$$ fixed (e.g. a new experiment with the same regressors), then the LASSO will not be unstable like making a different coefficient turn zero. For the 2d case it are random changes in the regressors $$X$$ (which may change the angle of the ellipse) that can cause instabilities.

In the question Does LASSO suffer from the same problems stepwise regression does? there is an example of an unstable lasso where entirely different coefficients from the true coefficients are selected. I will have to check whether this selection will vary only due to the variations in $$X$$ or whether in the higher dimensionality case, also changes in $$y$$ can cause the instabilities.

I have a minor follow-up question regarding the correlation between two variables. It occurred to me that when two variables are correlated, the thin and long ellipse shown in the left figure of your answer may also span from the bottom left to the top right. In such cases, would the coefficient of LASSO also be insensitive to small changes in the value of X?

I believe that what people mean with 'the instability of LASSO' is when small changes (in that ellipse) change the solution by a lot. One time you may have coefficient $$\beta_1$$ another time you may have coefficient $$\beta_2$$. That occurs when that ellipse is crossing both the $$\beta_1$$ axis and the $$\beta_2$$ axis at nearly the same distances.

For particular angles and positions of the ellipse the situation does not occur, but in practice variables can play similar roles such that the ellipse is roughly at -45° and centered around $$\beta_1 = \beta_2$$ (or anywhere such that it is crossing both axes at the same time will work).

The ellipse at a different angle 45° from bottom left to top right, that happens when two variables are negatively correlated. It will depend on the situation, but if at the same time the center of the ellipse is around $$\beta_1 = -\beta_2$$ (or anywhere away from the line $$\beta_1 =\beta_2$$ that would make the ellipse cross the origin instead of both axes) then you get similar instabilities.

• Thank you, Sextus! I've never seen this visualization before, and it's very insightful. I appreciate you sharing it with me.
– Alex
Mar 28, 2023 at 1:15
• When dealing with correlated variables, it can be challenging to determine which ones to retain in practical terms. Do you have any insight on how to identify which variables to keep?
– Alex
Mar 28, 2023 at 1:25
• @Alex the actual regression and selection itself is not so important. It is most important that you ensure that the training data and test data capture the final application well. A trained model is, in the end, just an abstract image of the previously seen data. How well it performs depends on how well the previous data is the same as the new data. The choice between ridge and lasso depends on how you want to incorporate bias to reduce influences of noise from the training phase. Lasso reduces more the number of (potentially superfluous) variables, ridge reduces more the size of variables. Mar 28, 2023 at 6:01
• In the linked question stats.stackexchange.com/questions/447684 I gave an example with red wine and white wine, which may correlate in most data sets and gives problems if you encounter a data set where they don't correlate. The problem is one of extrapolation to different settings. Mar 28, 2023 at 6:06
• Thank you, @Sextus, for your helpful response. I have a minor follow-up question regarding the correlation between two variables. It occurred to me that when two variables are correlated, the thin and long ellipse shown in the left figure of your answer may also span from the bottom left to the top right. In such cases, would the coefficient of LASSO also be insensitive to small changes in the value of X? I would appreciate it if you could confirm whether my understanding is correct.
– Alex
Mar 28, 2023 at 23:39