6
$\begingroup$

Has anyone ever applied a ridge regression on a model subset selected from a cross validated lasso? In other words, take a data set with p features and run lasso, grid searched to find optimal penalty parameters. Then record which features were dropped, and run only those features in a ridge regression. This approach seems similar to the "relaxed lasso" suggested by Meinhausen (2007) and clarified in the CV thread.

The only result on using ridge after lasso in literature I could find is this theoretical paper.

My intuition is that if relaxed lasso's objective is to split the separate variable selection and variable shrinkage, then why not go with ridge regression in the second pass? This will ensure that there isn't any variable selection in the second pass, whereas re-running lasso could result in additional feature drops.

$\endgroup$
2
  • 3
    $\begingroup$ See this thread: stats.stackexchange.com/questions/326427 -- some parts of my answer there are relevant. I used ridge after elastic net (ridge+lasso) as an attempt to make something like "relaxed elastic net". $\endgroup$
    – amoeba
    Jan 9, 2019 at 20:49
  • 3
    $\begingroup$ In this interesting paper De Mol 2009 the authors use a combination of elastic-net followed by ridge regression (which they simply refer to as regularized least squares) in order to select nested list of genes. The underlying idea is that the second ridge step should cope with the well-known over-shrinking effect of naïve elastic-net. $\endgroup$
    – fsamu
    Jan 9, 2019 at 21:20

1 Answer 1

1
$\begingroup$

This will ensure that there isn't any variable selection in the second pass, whereas re-running lasso could result in additional feature drops.

When performing relaxed LASSO then the point is to reduce the shrinking from a LASSO regression that is used for feature selection. For this purpose the second re-run with LASSO will be with a less severe penalty and will not result in additional feature drops.

Why a second lasso regression instead of a second ridge regression? Lasso is less severe in shrinking parameter estimates than ridge regression, although it also depends on the used penalty coefficients.

While the magnitude of the shrinking is more or less the same (as it can be regulated with the penalty coefficients), there are some qualitative differences (see also the different shapes of priors used by LASSO and ridge regression).

  • The prior for LASSO will drop of less quickly in longer ranges and penalize parameter estimates with different magnitudes more similar.

  • Ridge regression will penalize larger parameter estimates more severely and sort of pulls the estimates together to similar magnitudes.

Yet, possibly there is not much difference between the two situations when the regularisation is in the regime where the parameter estimates are all non-zero. Then the use of a second LASSO might possibly be still advantageous over a second ridge regression for computational reasons. The computations from the first regression, like LARS, can be quickly used in the second step, and the path is also a straight line that is easily computed.

lasso and ridge regression priors Image: comparison of priors for LASSO (Laplace distributed prior) and ridge regression (Gaussian distributed prior). The scales can differ when regularisation parameters are changed, but the shapes are the same. The Laplace distribution (for LASSO) is more pointy near zero, but has longer tails far away from zero. The Gaussian distribution (for ridge regression) has a more blunt peak at zero and places more focus on smaller tails, leading to smaller parameter estimates, but not neccesarily close to zero.

$\endgroup$
3
  • $\begingroup$ Using relaxed lasso means that you've given up on model calibration-in-the-small. $\endgroup$ Dec 27, 2023 at 10:49
  • $\begingroup$ @FrankHarrell this calibration-in-the-small versus calibration-in-the-large sound like interesting concepts that I haven't heard of before and had to look up with an internet search and a lot of the links direct back to you. Do you have a good pointer towards a reference that provides an introduction to these concepts? $\endgroup$ Dec 27, 2023 at 12:14
  • $\begingroup$ hbiostat.org/rmsc/validate plus several good papers related to the Brier score that I wish I had referenced there. Cal-in-the-small refers to the accuracy of probability forecasts at each level of predicted probability that occurs in the data. $\endgroup$ Dec 27, 2023 at 12:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.