I once heard a method of using the lasso twice (like a double-lasso) where you perform lasso on the original set of variables, say S1, obtain a sparse set called S2, and then perform lasso again on set S2 to obtain set S3. Is there a methodological term for this? Also, what are the advantages of doing lasso twice?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
1
Yes, the procedure you are asking (or thinking of) is called the relaxed lasso.
The general idea is that in the process of performing the LASSO for the first time you are probably including "noise variables"; performing the LASSO on a second set of variables (after the first LASSO) gives less competition between variables that are "real competitors" to being part of the model and not just "noise" variables. Technically, what this methods aims to is to overcome the (known) slow convergence of the LASSO in datasets with large number of variables.
You can read more about it on the original paper by Meinshausen (2007).
I also recommend section 3.8.5 on the Elements of Statistical Learning (Hastie, Tibshirani & Friedman, 2008), which gives an overview of other very interesting methods for performing variable selection using the LASSO.
-
$\begingroup$ Thanks! I'll definitely look into Meinshausen's article. $\endgroup$– BstatCommented Sep 26, 2012 at 18:01
$\begingroup$
$\endgroup$
0
The idea is to separate the two effects of lasso
- Variable selection (i.e., many, even most, $\beta$s are zero)
- Coefficient shrinkage (i.e., even non-zero $\beta$s are smaller, in absolute value, than in unpenalised regression). This is often a good thing even without selection because you avoid over-fitting.
If you have many variables ($p >\!\!> n$), and are running lasso, then you want to have a large penalty to select a small number of variables. However, this penalty might shrink the selected variables too much (you are under-fitting).
The idea of relaxed lasso is that you separate the two effects: you use a high penalty on the first pass to select variables; and a smaller penalty on the second pass to shrink them by a smaller amount.
The original paper (as linked by Néstor) gives more detail.
