2
$\begingroup$

What are a priori advantages of Lasso regularization for linear regression models, over many other heuristically-justifiable methods that both regularize the problem and perform variable selection?

For instance, what would be advantages of Lasso over simply performing ridge regression and then dropping the variables (terms) that have their linear coefficients falling below a certain small pre-specified threshold in absolute values (possibly with re-running the ridge regression again, after these variables are dropped)?

The heuristic motivation for Lasso, usually illustrated by a diagram showing the extremum of a quadratic function in $R^2$ achieved at the vertex of the $||x||_1<t$ constraint region, certainly reads appealing. However, do we have a proof that performing such regularization would ensure some useful property of the resulting solution, better than some/any other regularization methods?

$\endgroup$

1 Answer 1

5
$\begingroup$

You can see discussion about this in any textbook eg Elements of Statistical Learning. I would say those heuristics are hard to justify (ridge regression + dropping least significant)

roughly speaking ridge regression and lasso have different prior beliefs about the data.

ridge regression: the inputs are correlated,common signal + independent identically distributed noise. Averaging the inputs (as is done by ridge regression) therefore increases the signal to noise ratio. It should be clear that dropping variables is a bad idea, since the average of more variables is more accurate than one with less. an example might be exam results. However, if some inputs have more noise than others there will be a diminishing return from adding more variables to the average.

lasso regression: the high dimensional data have a fraction of important variables and the rest are noise. I believe this regularisation arose from gene microarray studies where only a handful of genes impact a given disease. Another example might be hierarchical data (eg school district-> school -> class), where assuming a given level of the hierarchy is relevant to the dependent variable, lasso will select that, whereas ridge will have some combination with the lower levels too.

Which of these best describe your data set will affect which of the 2 regularisations perform better in the ideal case, but often there is little difference in practise.

$\endgroup$
5
  • 1
    $\begingroup$ It is also important to note that lasso’s ability to do variable selection is largely a mirage. I say that because even if there are no collinearities among the Xs the probability that lasso chooses the “right” variables is zero. See for example fharrell.com/talk/stratos19 $\endgroup$ Commented Jan 28 at 14:34
  • $\begingroup$ @seanv507: Thank you for your input! Please correct me if I am wrong, what you are saying is that ridge regression and Lasso solve different problems (perhaps, having different assumptions about the statistical properties of the noise). Consider the problem you formulated as one to be solved by Lasso - "lasso regression: the high dimensional data have a fraction of important variables and the rest are noise". $\endgroup$
    – AL1117
    Commented Jan 29 at 5:50
  • $\begingroup$ @seanv507: (...continued) My question applies namely to this problem, for which the candidate solution can be obtained by using Lasso or by any other linear regression method, which also performs the variable selection (e.g. the trivial modification of the ridge regression described in my original question). What is the meaningful metric of the quality of the solution that Lasso regression will guarantee to be better that the same metric applied to solutions obtained by other methods? Thank you! $\endgroup$
    – AL1117
    Commented Jan 29 at 6:00
  • $\begingroup$ @FrankHarrell : Thank you for the intriguing comment and the link to two versions of the slides and the video of your talk. Which of these documents are the best presentation of the idea of your comment - the most uptodate slides ( hbiostat.org/talks/taiwan.pdf ) or something else? Thank you! $\endgroup$
    – AL1117
    Commented Jan 29 at 6:09
  • $\begingroup$ That link is good; also see hbiostat.org/bbr/hdata $\endgroup$ Commented Jan 29 at 13:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.