# Optimal penalty selection for lasso

Are there any analytical results or experimental papers regarding the optimal choice of the coefficient of the $\ell_1$ penalty term. By optimal, I mean a parameter that maximizes the probability of selecting the best model, or that minimizes the expected loss. I am asking because often it is impractical to choose the parameter by cross-validation or bootstrap, either because of a large number of instances of the problem, or because of the size of the problem at hand. The only positive result I am aware of is Candes and Plan, Near-ideal model selection by $\ell_1$ minimization.

-
Are you aware of papers establishing consistency results for the lasso? Knight & Fu (2000), Yu & Zhao (2006), and various papers by Meinshausen. –  cardinal May 7 '11 at 23:27
Yes, but my question in not about asymptotic consistency, which is the subject of the papers you mentioned. –  gappy May 8 '11 at 11:09
Those papers are (primarily) about model selection consistency, which I'd say is very related to the question you asked. :) –  cardinal May 8 '11 at 13:05

I take it that you are mostly interested in regression, as in the cited paper, and not other applications of the $\ell_1$-penalty (graphical lasso, say).

I then believe that some answers can be found in the paper On the “degrees of freedom” of the lasso by Zou et al. Briefly, it gives an analytic formula for the effective degrees of freedom, which for the squared error loss allows you to replace CV by an analytic $C_p$-type statistic, say.

Another place to look is in The Dantzig selector: Statistical estimation when p is much larger than n and the discussion papers in the same issue of Annals of Statistics. My understanding is that they solve a problem closely related to lasso regression but with a fixed choice of penalty coefficient. But please take a look at the discussion papers too.

If you are not interested in prediction, but in model selection, I am not aware of similar results. Prediction optimal models often result in too many selected variables in regression models. In the paper Stability selection Meinshausen and Bühlmann presents a subsampling technique more useful for model selection, but it may be too computationally demanding for your needs.

-
(+1) All three of those papers are worth a careful reading for those interested in this subject. The Dantzig selector paper has some very nice math; however, I have not seen it get much traction in applications, nor do I expect it to. I think, among other things, the very noisy regularization paths make people nervous and so, with no obvious benefit over the lasso, make it a hard sell. –  cardinal May 8 '11 at 13:08