10
$\begingroup$

I am looking forward to learning more about the regularized regression techniques like Ridge and Lasso regression. I would like to know what can be achieved by using these techniques when compared to linear regression model. Also in what situation we should adopt these techniques. And what makes these two techniques different. I am looking to understand the concept and maths behind these techniques. I would request to share your valuable knowledge.

$\endgroup$
3
  • $\begingroup$ The corresponding chapters in this leading (and freely downloadable) textbook offer good insights: www-bcf.usc.edu/~gareth/ISL $\endgroup$ Commented Dec 15, 2016 at 7:15
  • $\begingroup$ @ChristophHanck, that link doesn't work now. Will you be able to update the msg with a new link? $\endgroup$ Commented Jun 22, 2023 at 10:54
  • $\begingroup$ This will be statlearning.com (google "introduction to statistical learning" in case of further URL changes). $\endgroup$ Commented Jun 22, 2023 at 12:13

3 Answers 3

19
$\begingroup$

In short, ridge regression and lasso are regression techniques optimized for prediction, rather than inference.

Normal regression gives you unbiased regression coefficients (maximum likelihood estimates "as observed in the data-set").

Ridge and lasso regression allow you to regularize ("shrink") coefficients. This means that the estimated coefficients are pushed towards 0, to make them work better on new data-sets ("optimized for prediction"). This allows you to use complex models and avoid over-fitting at the same time.

For both ridge and lasso you have to set a so-called "meta-parameter" that defines how aggressive regularization is performed. Meta-parameters are usually chosen by cross-validation. For Ridge regression the meta-parameter is often called "alpha" or "L2"; it simply defines regularization strength. For LASSO the meta-parameter is often called "lambda", or "L1". In contrast to Ridge, the LASSO regularization will actually set less-important predictors to 0 and help you with choosing the predictors that can be left out of the model. The two methods are combined in "Elastic Net" Regularization. Here, both parameters can be set, with "L2" defining regularization strength and "L1" the desired sparseness of results.

Here you find a nice intro to the topic: http://scikit-learn.org/stable/modules/linear_model.html

$\endgroup$
2
  • 1
    $\begingroup$ Could you give more details about the 2 meta-parameters LASSO uses? I searched around and it seems that LASSO uses only 1 $\endgroup$
    – user152503
    Commented Jun 27, 2017 at 19:57
  • $\begingroup$ Thanks for raising my awareness to this point. I've previously mixed up "LASSO" with the more general "Elastic Net". See correction above. $\endgroup$
    – mzunhammer
    Commented Jun 28, 2017 at 11:08
3
$\begingroup$

Even though the linear model may be optimal for the data given to create the model, it is not necessarily guaranteed to be the best model for predictions on unseen data

If our underlying data follows a relatively simple model, and the model we use is too complex for the task, what we are essentially doing is we are putting too much weight on any possible change or variance in the data. Our model is overreacting and overcompensating for even the slightest change in our data. People in the field of statistics and machine learning call this phenomenon overfitting. When you have features in your dataset that are highly linearly correlated with other features, turns out linear models will be likely to overfit.

Ridge Regression, avoids over fitting by adding a penalty to models that have too large coefficients.

$\endgroup$
1
  • $\begingroup$ Well, yes, but ridge regression is an alternative estimator for a linear model $\endgroup$ Commented Jul 7, 2018 at 20:29
0
$\begingroup$

It all depends on the kind of problem you are dealing with. On a first glance, deciding upon a model of choice can be tricky.I think, first you should understand the dataset and how features interact with each other and come up to a good representation of your dataset to be used for modelling then comes your model of choice.

suppose you have a high dimensionality and high correlation in your dataset, then you would want to prefer L1(lasso) regularisation since it penalises less important features more and makes them zero which gives you the benefit of algorithmic feature selection and would make robust predictions than L2(ridge) regularisation but sometimes it can remove certain signals from the model even when they have information so it should be used carefully.

L2 regularisation handles the model complexity by focusing more on the important features which contribute more to the overall error than the less important features. But still, it uses information from less important features in the model. Different features contribute differently to the overall error and naturally our quest is to focus more on the important features which contribute more to the error than less important ones which can be handles with L2(ridge) regularisation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.