I'm just starting in machine learning and I can't figure out how does lasso method find which features are redundant to shrink their coefficients to zero?

  • 3
    $\begingroup$ Possible duplicate of Why L1 norm for sparse models $\endgroup$ Aug 10 '18 at 18:07
  • $\begingroup$ Your mistaken - lasso doesnt find the redundant features - it just shrinks features to zero somewhat arbitrarily - you need.to use another method like cross validation or scoring to determine which features are best $\endgroup$ Aug 25 '18 at 9:59

There are many ways to think about regularization. I find the restricted optimization formulation to be quite intuitive.

$$\hat\beta = \arg\min_\beta \|\bf y - X\bf\beta\|$$ $$\text{subject to } \|\bf \beta\| \leq \lambda_\star$$

Usually in the first line, we use the L2 norm (squared), which corresponds to Ordinary Least Squares estimates. The restriction gives a way of "shrinking" the estimates back to the origin.

If an L2 norm is used in the second line, we have Ridge Regression, which effectively pulls the OLS estimates back towards the origin. Useful in many situations, but not very good at setting estimates equal to 0.

If an L1 norm is used in the second line, we get LASSO. This is good for "feature selection" since it is able to effectively set inert or nearly-inert features to 0.

The following image illustrates this nicely in two dimensions.

enter image description here

Figure from Elements of Statistical Learning by Hastie, Tibshirani, and Friedman

  • $\begingroup$ Can you include some captions on the image, for example L1 (left) and L2 (right)? $\endgroup$
    – Jon
    Aug 13 '18 at 16:21
  • 2
    $\begingroup$ @Jon Good idea. How does that look? $\endgroup$
    – knrumsey
    Aug 13 '18 at 21:49

how does lasso method find which features are redundant to shrink their coefficients to zero?

The Lasso method on its own does not find which features to shrink. Instead, it is a combination of Lasso and Cross Validation (CV) which allows us to determine the best Lasso parameter $\lambda$.

Different $\lambda$ may shrink different parameters to zero, and choosing the right $\lambda$ can sometimes be easy (e.g.) in the graphic below, and sometimes be difficult/subjective task.

A good introduction to combining Lasso and cross validation is provided by the inventor of the Lasso, Robert Tibshirani, pages 15 and 16 here. Also here: Lasso cross validation and here

enter image description here

What happens when features are correlated

The case of correlated features and Lasso is more subtle and is well explained here: How does LASSO select among collinear predictors?.

But in practice there is some arbitrariness in which features are included in the model and when/why. Quoting EdM

In practice, the choice among correlated predictors in final models with either method is highly sample dependent, as can be checked by repeating these model-building processes on bootstrap samples of the same data. If there aren't too many predictors, and your primary interest is in prediction on new data sets, ridge regression, which tends to keep all predictors, may be a better choice.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.