I know that LASSO penalizes certain coefficents to zero by taking absolut value. However, ridge makes penalty by taking square instead. I am wondering why this difference forbid ridge from setting coefficient to zero like LASSO does. Can anyone share some comment? Thank a lot.
Quoted from Wikipedia:
LASSO AND RIDGE have different constraints: $||\beta||_1\leq t$ for LASSO and $||\beta||_2^2\leq t$ for RIDGE. The constraint region defined by the $\ell ^1$ norm is a square rotated so that its corners lie on the axes (in general a cross-polytope), while the region defined by the $\ell ^2$ norm is a circle (in general an n-sphere), which is rotationally invariant and, therefore, has no corners. A convex object that lies tangent to the boundary is likely to encounter a corner (or a higher-dimensional equivalent) of a hypercube, for which some components of $\beta$ are identically zero, while in the case of an n-sphere, the points on the boundary for which some of the components of $\beta$ are zero are not distinguished from the others and the convex object is no more likely to contact a point at which some components of $\beta$ are zero than one for which none of them are.