# Lasso regression feature selection

I have been reading many articles on LASSO regression, and everyone claims that LASSO address multicollinearity showing contour plots of cost function touching the corner of the diamond(x1+x2). In the first place, I didn't understand how does this phenomenon translate to addressing multicollinearity when there are many groups of correlated variables in the data.

Verbatim from one of the research paper: "• If there are grouped variables (highly correlated between each other) LASSO tends to select one variable from each group ignoring the others".

• It's a bit unclear what you are asking. When you say multicollinearity, you must mean multi-correlation, since lasso still fails to find a unique solution in the case where there is a strict linear dependence between predictors. With that said, what exactly do you mean by "address multi-correlation""? Would you like an explanation for why the quote is true? – Matthew Drury Jan 20 '18 at 18:08
• yeah please advise why that quote is true. – AjitKrish Jan 20 '18 at 18:11
• That quote doesn't indicate that the LASSO handles correlated variables. It says the opposite; the behaviour of the LASSO is a tendency to select 1 variable from the group of correlated variables. We can interpret that behaviour as saying that LASSO would consider the effects of the non-selected variables as being 0, or that only that one variable has any effect. That is inconsistent with the variables being correlated. – Gavin Simpson Jan 20 '18 at 22:30

Everyone claims that LASSO address multicollinearity

Lasso regression allows you to set one or many of your feature coefficients to be exactly equal to zero by varying the parameter $\lambda$. In cases of multi-correlation, i.e. many features are correlated with each other, this can be useful as the Lasso regression will set some of them to zero and leave the others to do their job, hence reducing the number of correlated variables.

I didn't understand how does this phenomenon translate to addressing multicollinearity when there are many groups of correlated variables in the data

As @Gavin Simpson points out, Lasso will have a tendency to select 1 variable from a group of correlated variables - but again this will depend on the value of $\lambda$ chosen and on the specifics of your dataset - it is not a rule or a certainty.

### An example

An example in 2D as it is hard to visualizing anything more - where $x_1$ and $x_2$ are highly correlated, you see that from the shape of the OLS contour plot on the bottom right At some point, there is a value of $\lambda$ where the lasso solution (the red dot) is equal to 0 for one coefficient ($\beta_1$) and $-2.5$ for the other coefficient ($\beta_2$) - so you could say that this particular value of $\lambda$ allows to select only one feature from a group (of 2) correlated features. The same applies in higher dimensions

Code to generate the figure can be found here

As has been pointed out by others, Lasso can set coefficients to zero, thus in essence excluding them from the model. (Technically, the coefficient still affects the model since its initial inclusion affects how the other variables' coefficients are shrunk. More on this shrinkage soon.)

Furthermore, even when the coefficients are not set to zero, they tend to be shrunk towards zero. This can be seen by looking at what the ordinary least squares (OLS) estimated coefficients have minimized and compare it to the Lasso solution. In the notation below, $\beta=\beta_1, \ldots, \beta_p$ is a vector of $p$ coefficients, $\beta_0$ is an intercept, $y_i$ is the outcome of individual $i=1,\ldots,n$ and $x_{ki}$ is the value of covariate $k=1,\ldots,p$ for individual $i$.

$\hat{\beta}^{OLS} = \underset{\beta}{\arg\min}\left\{\underset{i=1}{\overset{n}{\sum}}\left( y_i - \beta_0 - \underset{k=1}{\overset{p}{\sum}}\beta_k x_{ki}\right)^2 \right\}$

That is, the OLS solution tries to minimize the mean squared error of prediction of $y_i$. If the covariates are highly correlated, it is known that this can lead to very instable estimates of the corresponding $\beta_k$. This high variance of the estimates can lead to $\beta_k$ to often be estimated far too large (in either direction). To combat that, the Lasso solution finds the solution that minimizes the mean squared error, penalized by the sum of the absolute value of the estimated coefficients.

$\hat{\beta}^{Lasso} = \underset{\beta}{\arg\min}\left\{\underset{i=1}{\overset{n}{\sum}}\left( y_i - \beta_0 - \underset{k=1}{\overset{p}{\sum}}\beta_k x_{ki}\right)^2 + \lambda \underset{k=1}{\overset{p}{\sum}} \vert\beta_k\vert \right\}$, for some $\lambda \geq 0.$

This means that solutions that have high values of the estimated coefficients are not as often selected as they would be in the OLS case. This leads to less variance in the estimated coefficients.

The cost of this adjustment is bias, as while $E\left(\hat{\beta}^{OLS}\right)=\beta$ even in the presence of multicollinearity, $E\left(\hat{\beta}^{Lasso}\right)\neq\beta$. Thus the choice of using the Lasso over OLS follows the usual bias-variance trade-off.

To summarize, the aspect of multicollinearity that Lasso combats is the high variance of the estimated coefficients that results from multicollinearity. It does this by shrinking coefficients towards zero, reducing the variance but introducing bias. Furthermore, it can set coefficients to exactly zero. This is unlike the very related Ridge regression (Hoerl & Kennard, 1970), where the coefficients are shrunk but can't be set to zero.

If you want more details about Lasso, Ridge and similar shrinkage techniques, I recommend Hastie et al. (2009).

References:

Hastie, T., Tibshirani, R., and Friedman, J. (2009). The elements of statistical learning. Springer series in statistics. Springer, New York, 11th printing, 2nd edition.

Hoerl, A. E., and Kennard, R. W. (1970). Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1), 55-67.