I've been to a few statistics talks recently on the Lasso (regularization) and a point that keeps coming up is that we do not really understand why the Lasso works or why it works so well. I am wondering what this statement is referring to. Obviously I understand why the Lasso works technically, by way of prevention of overfitting by shrinkage of parameters, but I am wondering if there is a deeper meaning behind such a statement. Does anyone have any ideas? Thanks!
There is sometimes a lack of communication between working statisticians and the learning theory community that study the foundations of methods like the lasso. The theoretical properties of the lasso are actually very well understood.
This document has a summary in Section 4 of many of the properties it enjoys. The results are quite technical, but essentially:
- It recovers the true support (set of non-zero entries) of a sparse weight vector under some mild assumptions, for large enough datasets, with high probability.
- It converges to the correct weight vector at the optimal rate as the sample size increases, as long as the columns of $X$ are not too correlated.
There's the problem of sign recovery of model selection consistency (which has answered by statisticians), and
there's the problem of inference (constructing good confidence intervals for the estimates), which is till a topic of research.
Most of the work is done by statisticians rather than "the learning theory community".