10
$\begingroup$

I've been to a few statistics talks recently on the Lasso (regularization) and a point that keeps coming up is that we do not really understand why the Lasso works or why it works so well. I am wondering what this statement is referring to. Obviously I understand why the Lasso works technically, by way of prevention of overfitting by shrinkage of parameters, but I am wondering if there is a deeper meaning behind such a statement. Does anyone have any ideas? Thanks!

$\endgroup$
  • 1
    $\begingroup$ Define "works". Works to do what, exactly? Works to increase sparsity? Works to prevent overfitting? Works to produce reasonable statistical tests? -- Or to put it another way, what would "not working" mean in this context? -- As you can see from the comments on the current answer, there's some confusion as to what you're after. $\endgroup$ – R.M. Mar 30 '17 at 14:52
  • $\begingroup$ @R.M., you are actually just rephrasing the OP, IMHO. The OP is probably after the same unknown as the one you have trouble to identify. $\endgroup$ – Richard Hardy Mar 30 '17 at 15:06
  • 1
    $\begingroup$ @RichardHardy I see how that might be the case, but if so, I would hope that the OP could at least expand upon the context in those statistics talks in which the point came up, to hopefully help us focus in on what those speakers might have been thinking. $\endgroup$ – R.M. Mar 30 '17 at 15:15
  • $\begingroup$ @R.M., good then. $\endgroup$ – Richard Hardy Mar 30 '17 at 15:22
11
$\begingroup$

There is sometimes a lack of communication between working statisticians and the learning theory community that study the foundations of methods like the lasso. The theoretical properties of the lasso are actually very well understood.

This document has a summary in Section 4 of many of the properties it enjoys. The results are quite technical, but essentially:

  • It recovers the true support (set of non-zero entries) of a sparse weight vector under some mild assumptions, for large enough datasets, with high probability.
  • It converges to the correct weight vector at the optimal rate as the sample size increases, as long as the columns of $X$ are not too correlated.
$\endgroup$
3
$\begingroup$

If by understanding why Lasso works, you mean understanding why it performs feature selection (i.e. setting weights for some features to exactly 0), we understand that very well:

Lasso regularization as Lagrangian optimization

$\endgroup$
  • 4
    $\begingroup$ Thanks for a nice illustration, but I suspect that is not the part the OP is interested in. Of course, it is up to the OP to clarify that. $\endgroup$ – Richard Hardy Mar 30 '17 at 14:15
  • $\begingroup$ I don't understand the point(s) of your diagram. $\endgroup$ – Michael Chernick Mar 30 '17 at 14:16
  • 7
    $\begingroup$ I’ve downvoted because this diagram has been around since at least Tibshirani's original lasso paper and does not help the question. We do understand very well why an $L_1$ penalty leads to sparsity in the standard lasso, but there's a lot more to the lasso than just that. There are questions of distributions of coefficients and hypothesis tests, modifying the penalty to force certain zero patterns, asymptotic results like irrepresentableness, performance when we plug in $\hat \lambda$ chosen via CV, and much more $\endgroup$ – jld Mar 30 '17 at 14:46
  • 4
    $\begingroup$ @Chaconne, your points form a great basis for an answer! $\endgroup$ – Richard Hardy Mar 30 '17 at 15:04
  • 1
    $\begingroup$ @Chaconne, it did seem to generate useful discussion though by identifying what we do understand about Lasso! $\endgroup$ – rinspy Mar 30 '17 at 15:38
2
$\begingroup$

There's the problem of sign recovery of model selection consistency (which has answered by statisticians), and

there's the problem of inference (constructing good confidence intervals for the estimates), which is till a topic of research.

Most of the work is done by statisticians rather than "the learning theory community".

$\endgroup$
  • $\begingroup$ How does this add to what was already given? $\endgroup$ – Michael Chernick Apr 4 '17 at 20:54
  • $\begingroup$ No one has mentioned the problem of inference here, which I believe is the reason why the claim ("it is not well-understood") was made in the first place. $\endgroup$ – GAO Zheng Apr 5 '17 at 16:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.