What happens to Lasso Regression when variables are collinear? How do we deal with it?

Question

I'm self-studying the Elements of Statistical Learning and I came across this question

I believe that rather than a single solution, there's now a manifold of solutions with $\hat{\beta}_j + \hat{\beta}_j^* = \hat{\beta}_j^{(original)} = a$.

From a pictoral perspective, originally we'd have the likelihood of the dataset intersect at a single point (see left): But if $\beta_2$ corresponded to a duplicated variable, then we'd have that the likelihood contour should perfectly line up with the edge of the diamond and there's no single optimal solution.

What is the lesson here? I'm not sure what I'm supposed to take away from this problem.

Furthermore, it seems like certain numerical optimization procedures for lasso regression such as least angles regression - which iterates one variable at a time - might permanently set one of the duplicated pair's coefficient to zero anyways?

Answer 1

I think that you get the take-home point quite well.

With collinear predictors, LASSO and other variable-selection methods necessarily make arbitrary choices about which to include. The perfect collinearity in this example is extreme, but the problem arises with less extreme multicollinearity. See this thread among many on this site; e.g., search for lasso bootstrap instability.