I am looking forward to learning more about the regularized regression techniques like Ridge and Lasso regression. I would like to know what can be achieved by using these techniques when compared to linear regression model. Also in what situation we should adopt these techniques. And what makes these two techniques different. I am looking to understand the concept and maths behind these techniques. I would request to share your valuable knowledge.
In short, ridge regression and lasso are regression techniques optimized for prediction, rather than inference.
Normal regression gives you unbiased regression coefficients (maximum likelihood estimates "as observed in the data-set").
Ridge and lasso regression allow you to regularize ("shrink") coefficients. This means that the estimated coefficients are pushed towards 0, to make them work better on new data-sets ("optimized for prediction"). This allows you to use complex models and avoid over-fitting at the same time.
For both ridge and lasso you have to set a so-called "meta-parameter" that defines how aggressive regularization is performed. Meta-parameters are usually chosen by cross-validation. For Ridge regression the meta-parameter is often called "alpha" or "L2"; it simply defines regularization strength. For LASSO the meta-parameter is often called "lambda", or "L1". In contrast to Ridge, the LASSO regularization will actually set less-important predictors to 0 and help you with choosing the predictors that can be left out of the model. The two methods are combined in "Elastic Net" Regularization. Here, both parameters can be set, with "L2" defining regularization strength and "L1" the desired sparseness of results.
Here you find a nice intro to the topic: http://scikit-learn.org/stable/modules/linear_model.html
Even though the linear model may be optimal for the data given to create the model, it is not necessarily guaranteed to be the best model for predictions on unseen data
If our underlying data follows a relatively simple model, and the model we use is too complex for the task, what we are essentially doing is we are putting too much weight on any possible change or variance in the data. Our model is overreacting and overcompensating for even the slightest change in our data. People in the field of statistics and machine learning call this phenomenon overfitting. When you have features in your dataset that are highly linearly correlated with other features, turns out linear models will be likely to overfit.
Ridge Regression, avoids over fitting by adding a penalty to models that have too large coefficients.