When to use Ridge regression and Lasso regression. What can be achieved while using these techniques rather than the linear regression model

Question

I am looking forward to learning more about the regularized regression techniques like Ridge and Lasso regression. I would like to know what can be achieved by using these techniques when compared to linear regression model. Also in what situation we should adopt these techniques. And what makes these two techniques different. I am looking to understand the concept and maths behind these techniques. I would request to share your valuable knowledge.

Answer 1

In short, ridge regression and lasso are regression techniques optimized for prediction, rather than inference.

Normal regression gives you unbiased regression coefficients (maximum likelihood estimates "as observed in the data-set").

Ridge and lasso regression allow you to regularize ("shrink") coefficients. This means that the estimated coefficients are pushed towards 0, to make them work better on new data-sets ("optimized for prediction"). This allows you to use complex models and avoid over-fitting at the same time.

For both ridge and lasso you have to set a so-called "meta-parameter" that defines how aggressive regularization is performed. Meta-parameters are usually chosen by cross-validation. For Ridge regression the meta-parameter is often called "alpha" or "L2"; it simply defines regularization strength. For LASSO the meta-parameter is often called "lambda", or "L1". In contrast to Ridge, the LASSO regularization will actually set less-important predictors to 0 and help you with choosing the predictors that can be left out of the model. The two methods are combined in "Elastic Net" Regularization. Here, both parameters can be set, with "L2" defining regularization strength and "L1" the desired sparseness of results.

Here you find a nice intro to the topic: http://scikit-learn.org/stable/modules/linear_model.html

Answer 2

Even though the linear model may be optimal for the data given to create the model, it is not necessarily guaranteed to be the best model for predictions on unseen data

If our underlying data follows a relatively simple model, and the model we use is too complex for the task, what we are essentially doing is we are putting too much weight on any possible change or variance in the data. Our model is overreacting and overcompensating for even the slightest change in our data. People in the field of statistics and machine learning call this phenomenon overfitting. When you have features in your dataset that are highly linearly correlated with other features, turns out linear models will be likely to overfit.

Ridge Regression, avoids over fitting by adding a penalty to models that have too large coefficients.

Answer 3

It all depends on the kind of problem you are dealing with. On a first glance, deciding upon a model of choice can be tricky.I think, first you should understand the dataset and how features interact with each other and come up to a good representation of your dataset to be used for modelling then comes your model of choice.

suppose you have a high dimensionality and high correlation in your dataset, then you would want to prefer L1(lasso) regularisation since it penalises less important features more and makes them zero which gives you the benefit of algorithmic feature selection and would make robust predictions than L2(ridge) regularisation but sometimes it can remove certain signals from the model even when they have information so it should be used carefully.

L2 regularisation handles the model complexity by focusing more on the important features which contribute more to the overall error than the less important features. But still, it uses information from less important features in the model. Different features contribute differently to the overall error and naturally our quest is to focus more on the important features which contribute more to the error than less important ones which can be handles with L2(ridge) regularisation.