1
$\begingroup$

In applying a penalty term with either $l_1$ or $l_2$ norm, why would the former result in variable selection but not the latter?

$\endgroup$
1
3
$\begingroup$

The constrained region for l1 is diamond shaped(pointed at axis) while for l2 it is round(in 2 dimensions, similary rhomboid and sphere in case of higher dimensions). The residual sum of squares have elliptical contours, centered at full least square estimate. Both methods find first point where elliptical contours hit the constrained region. Since rhobus has corners at axis one is more likely to find a solution where one of the coefficients is 0.

Please refer to Elements o Statistical Learning by Tibshirani and Hastie(second edition page 71).

Image from ELS

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.