Relaxed LASSO - disambiguation

Question

In the book Statistical Learning with Sparsity - The Lasso and Generalizations by Hastie, Tibshirani and Wainwright we can read the following paragraph (p.12):

(...) lasso sets two of the five coefficients to zero, and tends to shrink the coefficients of the others toward zero relative to the full least-squares estimate. In turn, the least-squares fit on the subset of the three predictors tends to expand the lasso estimates away from zero. The nonzero estimates from the lasso tend to be biased toward zero, so the debiasing (...) can often improve the prediction error of the model. This two-stage process is also known as the relaxed lasso (Meinshausen 2007).

They explicitly cite the work by Meinshausen about relaxed lasso however my impression is that the Meinshausen's relaxed lasso is (in short) about performing lasso twice, not lasso+least squares.

Can anyone please clarify that? Is this some sort of shortcut by Hastie & co.?

Answer 1

Have a close look at the discussion following Definition 1 in the paper you linked. When Hastie, Tibshirani and Wainwright write "This two-stage process is also known as the relaxed lasso", they seem to be referring to the case $\phi = 0$.

Here is the relevant quote from Meinshausen:

The case of $\phi = 0$ needs special consideration, as the definition above would produce a degenerate solution. In the following, we define the relaxed Lasso estimator for $\phi = 0$ as the limit of the above definition for $\phi \rightarrow 0$. In this case, all coefficients in the model $M_\lambda$ are estimated by the OLS-solution.

The table of regression coefficients in the Hastie et al book (Table 2.2 Results from analysis of the crime data) uses the $\phi = 0$ version of the relaxed lasso, i.e. they run the lasso to get the set of predictors $M_\lambda$, and then they run regular OLS using only the set of predictors chosen by the lasso.