I'm looking for resources on lasso regression at the undergraduate mathematics level. All I can find is a lot of complex texts on variable selection, concentration inequalities, etc. I would like to understand more about how the lasso selects variables and works. I would also like to learn more about when the lasso should be used (particularly in comparison to other regularisation techniques), and how it compares to traditional model selection. Could anyone point me to some good notes on this?
$\begingroup$
$\endgroup$
3
-
1$\begingroup$ I think the books from Rob Tibshirani and coauthors may be helpful, including "Introduction to Statistical Learning", "Elements of statistical learning", and "Statistical learning with sparsity" $\endgroup$– BenCommented Jul 20, 2022 at 16:06
-
2$\begingroup$ @Ben: do you want to post your comment(s) as an answer? Better to have a short answer than no answer at all. Anyone who has a better answer can post it. $\endgroup$– Stephan KolassaCommented Jul 20, 2022 at 16:07
-
$\begingroup$ Find the book "Introduction to Statistical Learning" here: statlearning.com Ch. 6.2.2 $\endgroup$– PeterCommented Jul 20, 2022 at 17:14
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
I think the suggestions in the comments above will all be good references:
- Introduction to Statistical Learning (ISL or ISLR, see statlearning.com) gives a "broad and less technical treatment of key topics in statistical learning". This should be friendly for an undergraduate and has emphasis on applying various methods with R.
- Elements of Statistical Learning (ESL, see ElemStatLearn) is a very popular, more mathematical exposition of a range of statistical learning methods.
- Statistical learning with sparsity (SLS, see StatLearnSparsity) is a more recent, more mathematical text. As well as the Lasso, it considers applications of L1 penalisation in a wide range of contexts, such as to Penalised Matrix Decompositions and structure estimation in graphical models. This contains more details on the various convex optimisation algorithms required to solve Lasso-type problems and a wealth of references.
The three references above all have Tibshirani and Hastie in their authoriship; they have similar perspectives but are pitched at different levels. In terms of more theoretical view, the undergraduate course I attended has a helpful webpage with comprehensive notes and further references; see Modern Statistical Methods lecture course by Rajen Shah.
I'd probably start with ESL; but move to ISL if you want something more applied, or move to SLS if you want to understand the method better theoretically.
This didn't really answer your question though: in brief I think one should use Lasso regression when the assumptions underlying the model are well satisfied: i.e. linear model, sparse coefficient vector; and when the sparsity aids interpretability. I don't know of resources giving comprehensive comparisons between using different regularisation techniques in practice - but perhaps someone else can suggest?