Say I want to estimate a large number of parameters, and I want to penalize some of them because I believe they should have little effect compared to the others. How do I decide what penalization scheme to use? When is ridge regression more appropriate? When should I use lasso?
$\begingroup$
$\endgroup$
2
-
1$\begingroup$ "Say I want to estimate a large number of parameters" this could be made more precise: What is the framework ? I guess it is linear regression? $\endgroup$– robin girardJul 28, 2010 at 5:59
-
$\begingroup$ You say "lasso vs ridge" as if they're the only two options - what about generalised double pareto, horseshoe, bma, bridge, among others? $\endgroup$– probabilityislogicApr 12, 2014 at 1:16
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
2
Keep in mind that ridge regression can't zero out coefficients; thus, you either end up including all the coefficients in the model, or none of them. In contrast, the LASSO does both parameter shrinkage and variable selection automatically. If some of your covariates are highly correlated, you may want to look at the Elastic Net [3] instead of the LASSO.
I'd personally recommend using the Non-Negative Garotte (NNG) [1] as it's consistent in terms of estimation and variable selection [2]. Unlike LASSO and ridge regression, NNG requires an initial estimate that is then shrunk towards the origin. In the original paper, Breiman recommends the least-squares solution for the initial estimate (you may however want to start the search from a ridge regression solution and use something like GCV to select the penalty parameter).
In terms of available software, I've implemented the original NNG in MATLAB (based on Breiman's original FORTRAN code). You can download it from:
http://www.emakalic.org/blog/wp-content/uploads/2010/04/nngarotte.zip
BTW, if you prefer a Bayesian solution, check out [4,5].
References:
[1] Breiman, L. Better Subset Regression Using the Nonnegative Garrote Technometrics, 1995, 37, 373-384
[2] Yuan, M. & Lin, Y. On the non-negative garrotte estimator Journal of the Royal Statistical Society (Series B), 2007, 69, 143-161
[3] Zou, H. & Hastie, T. Regularization and variable selection via the elastic net Journal of the Royal Statistical Society (Series B), 2005, 67, 301-320
[4] Park, T. & Casella, G. The Bayesian Lasso Journal of the American Statistical Association, 2008, 103, 681-686
[5] Kyung, M.; Gill, J.; Ghosh, M. & Casella, G. Penalized Regression, Standard Errors, and Bayesian Lassos Bayesian Analysis, 2010, 5, 369-412
-
8$\begingroup$ Could you be more specific on ridge vs lasso? Is automatic variable selection the only reason to prefer lasso? $\endgroup$– ChoggOct 3, 2017 at 17:43
-
1$\begingroup$ why lasso is not a good idea when there are correlated covariates? $\endgroup$ Sep 20, 2021 at 8:40
$\begingroup$
$\endgroup$
5
Ridge or lasso are forms of regularized linear regressions. The regularization can also be interpreted as prior in a maximum a posteriori estimation method. Under this interpretation, the ridge and the lasso make different assumptions on the class of linear transformation they infer to relate input and output data. In the ridge, the coefficients of the linear transformation are normal distributed and in the lasso they are Laplace distributed. In the lasso, this makes it easier for the coefficients to be zero and therefore easier to eliminate some of your input variable as not contributing to the output.
There are also some practical considerations. The ridge is a bit easier to implement and faster to compute, which may matter depending on the type of data you have.
If you have both implemented, use subsets of your data to find the ridge and the lasso and compare how well they work on the left out data. The errors should give you an idea of which to use.
-
12$\begingroup$ I don't get it - how would you know if your coefficients are laplace or normal distributed? $\endgroup$– ihadannySep 20, 2015 at 21:15
-
3
-
4$\begingroup$ @Hbar: " The regularization can also be interpreted as prior in a maximum a posteriori estimation method. ": could you please explain this part in more detail with mathematical symbols, or at least give a reference? Thanks! $\endgroup$– MathmathSep 17, 2017 at 17:19
-
2$\begingroup$ @ihadanny You most probably wouldn't know, and that's the point. You can only decide which one to keep a posteriori. $\endgroup$– FirebugOct 11, 2017 at 16:23
-
1$\begingroup$ @Archie There is a closed form solution for Ridge Regression just like OLS but LASSO have to be computed using optimization procedures. $\endgroup$– FerusJan 18, 2020 at 13:42
$\begingroup$
$\endgroup$
3
Generally, when you have many small/medium sized effects you should go with ridge. If you have only a few variables with a medium/large effect, go with lasso. Hastie, Tibshirani, Friedman
-
6$\begingroup$ But when you have a few variables, you may want to keep them all in your models if they medium/large effects, which will not be the case in lasso as it might remove one of them. Can you please explain this in detail? I feel when you have many variables we use Lasso to remove unnecessary variables and not ridge. $\endgroup$ Oct 11, 2017 at 16:20
-
$\begingroup$ When we have many small/medium sized effects (i.e. several features with similar coefficients showing that almost all of them somehow impact the response), ridge could be used (one reason is, with Lasso when the number of features go up, there are many corners in the constraint region and this could increase the overall chance to remove (potentially valuable) features). This is the theory! In reality, we do not know the true coefficients of features and cannot easily decide. Therefore, you could always try both approaches and use cross-validation/performance-evaluation to compare results. $\endgroup$ Feb 13, 2022 at 21:28
-
$\begingroup$ If you have only a few variables with a medium/large effect (i.e. actually a few of the features impact the response), you can go with Lasso. It will try to keep only the important ones. (here as well when the number of features is not large, Lasso constraint space does not have too many corners and generally you shouldn't worry about Lasso making your variable zero by mistake. As I said, however, usually it is best to try both approaches and compare. $\endgroup$ Feb 13, 2022 at 21:30