0
$\begingroup$

I know that LASSO penalizes certain coefficents to zero by taking absolut value. However, ridge makes penalty by taking square instead. I am wondering why this difference forbid ridge from setting coefficient to zero like LASSO does. Can anyone share some comment? Thank a lot.

$\endgroup$
  • $\begingroup$ You mean LASSO is artificially allowed to set coefficent to zero while ridge is not? I thought there are some mathematical reason behind it. $\endgroup$ – unicorn Dec 7 '18 at 9:24
2
$\begingroup$

Quoted from Wikipedia:

LASSO AND RIDGE have different constraints: $||\beta||_1\leq t$ for LASSO and $||\beta||_2^2\leq t$ for RIDGE. The constraint region defined by the $\ell ^1$ norm is a square rotated so that its corners lie on the axes (in general a cross-polytope), while the region defined by the $\ell ^2$ norm is a circle (in general an n-sphere), which is rotationally invariant and, therefore, has no corners. A convex object that lies tangent to the boundary is likely to encounter a corner (or a higher-dimensional equivalent) of a hypercube, for which some components of $\beta$ are identically zero, while in the case of an n-sphere, the points on the boundary for which some of the components of $\beta$ are zero are not distinguished from the others and the convex object is no more likely to contact a point at which some components of $\beta$ are zero than one for which none of them are.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.