Why not use Ridge after Lasso vs relaxed Lasso

Question

Has anyone ever applied a ridge regression on a model subset selected from a cross validated lasso? In other words, take a data set with p features and run lasso, grid searched to find optimal penalty parameters. Then record which features were dropped, and run only those features in a ridge regression. This approach seems similar to the "relaxed lasso" suggested by Meinhausen (2007) and clarified in the CV thread.

The only result on using ridge after lasso in literature I could find is this theoretical paper.

My intuition is that if relaxed lasso's objective is to split the separate variable selection and variable shrinkage, then why not go with ridge regression in the second pass? This will ensure that there isn't any variable selection in the second pass, whereas re-running lasso could result in additional feature drops.

Answer 1

This will ensure that there isn't any variable selection in the second pass, whereas re-running lasso could result in additional feature drops.

When performing relaxed LASSO then the point is to reduce the shrinking from a LASSO regression that is used for feature selection. For this purpose the second re-run with LASSO will be with a less severe penalty and will not result in additional feature drops.

Why a second lasso regression instead of a second ridge regression? Lasso is less severe in shrinking parameter estimates than ridge regression, although it also depends on the used penalty coefficients.

While the magnitude of the shrinking is more or less the same (as it can be regulated with the penalty coefficients), there are some qualitative differences (see also the different shapes of priors used by LASSO and ridge regression).

• The prior for LASSO will drop of less quickly in longer ranges and penalize parameter estimates with different magnitudes more similar.

• Ridge regression will penalize larger parameter estimates more severely and sort of pulls the estimates together to similar magnitudes.

Yet, possibly there is not much difference between the two situations when the regularisation is in the regime where the parameter estimates are all non-zero. Then the use of a second LASSO might possibly be still advantageous over a second ridge regression for computational reasons. The computations from the first regression, like LARS, can be quickly used in the second step, and the path is also a straight line that is easily computed.

Image: comparison of priors for LASSO (Laplace distributed prior) and ridge regression (Gaussian distributed prior). The scales can differ when regularisation parameters are changed, but the shapes are the same. The Laplace distribution (for LASSO) is more pointy near zero, but has longer tails far away from zero. The Gaussian distribution (for ridge regression) has a more blunt peak at zero and places more focus on smaller tails, leading to smaller parameter estimates, but not neccesarily close to zero.