In the book Statistical Learning with Sparsity - The Lasso and Generalizations by Hastie, Tibshirani and Wainwright we can read the following paragraph (p.12):

(...) lasso sets two of the five coefficients to zero, and tends to shrink the coefficients of the others toward zero relative to the full least-squares estimate. In turn, the least-squares fit on the subset of the three predictors tends to expand the lasso estimates away from zero. The nonzero estimates from the lasso tend to be biased toward zero, so the debiasing (...) can often improve the prediction error of the model. This two-stage process is also known as the relaxed lasso (Meinshausen 2007).

They explicitly cite the work by Meinshausen about relaxed lasso however my impression is that the Meinshausen's relaxed lasso is (in short) about performing lasso twice, not lasso+least squares.

Can anyone please clarify that? Is this some sort of shortcut by Hastie & co.?


1 Answer 1


Have a close look at the discussion following Definition 1 in the paper you linked. When Hastie, Tibshirani and Wainwright write "This two-stage process is also known as the relaxed lasso", they seem to be referring to the case $\phi = 0$.

Here is the relevant quote from Meinshausen:

The case of $\phi = 0$ needs special consideration, as the definition above would produce a degenerate solution. In the following, we define the relaxed Lasso estimator for $\phi = 0$ as the limit of the above definition for $\phi \rightarrow 0$. In this case, all coefficients in the model $M_\lambda$ are estimated by the OLS-solution.

The table of regression coefficients in the Hastie et al book (Table 2.2 Results from analysis of the crime data) uses the $\phi = 0$ version of the relaxed lasso, i.e. they run the lasso to get the set of predictors $M_\lambda$, and then they run regular OLS using only the set of predictors chosen by the lasso.

  • 3
    $\begingroup$ The bottom line is, it seems, that the quoted sentence from Hastie et al is not very precise. It would have been better to write that "This two-stage process" is one special case or maybe a particular application of a procedure "known as the relaxed lasso". $\endgroup$
    – amoeba
    Feb 5, 2018 at 13:26
  • 1
    $\begingroup$ So to get this straight: is a relaxed LASSO simply an OLS fit on the LASSO selected features? If so, should the sparsity of the model not remain the same? But if one fits glmnet models with relax=T and gamma=0, as described in cran.r-project.org/web/packages/glmnet/vignettes/relax.pdf, the resulting model ends up more sparse than the original LASSO fit. How is this possible actually? Or is that just due to a different lambda then being selected by cross validation (a higher one, as the relaxation reduced the bias on the coefficients)? $\endgroup$ Oct 12 at 15:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.