3
$\begingroup$

I'm self-studying the Elements of Statistical Learning and I came across this question enter image description here

I believe that rather than a single solution, there's now a manifold of solutions with $\hat{\beta}_j + \hat{\beta}_j^* = \hat{\beta}_j^{(original)} = a$.

From a pictoral perspective, originally we'd have the likelihood of the dataset intersect at a single point (see left): enter image description here But if $\beta_2$ corresponded to a duplicated variable, then we'd have that the likelihood contour should perfectly line up with the edge of the diamond and there's no single optimal solution.

What is the lesson here? I'm not sure what I'm supposed to take away from this problem.

Furthermore, it seems like certain numerical optimization procedures for lasso regression such as least angles regression - which iterates one variable at a time - might permanently set one of the duplicated pair's coefficient to zero anyways?

$\endgroup$

1 Answer 1

2
$\begingroup$

I think that you get the take-home point quite well.

With collinear predictors, LASSO and other variable-selection methods necessarily make arbitrary choices about which to include. The perfect collinearity in this example is extreme, but the problem arises with less extreme multicollinearity. See this thread among many on this site; e.g., search for lasso bootstrap instability.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.