Lasso and Ridge Regression: Variance and Bias

Question

I am studying about learning methods in statistics and the regression section explains that the main difference between Lasso and Ridge regressions is the formulation of the regularization term, which for Lasso is the ℓ1 norm, and for Ridge the ℓ2 norm.

Given that Lasso regression shrinks some of the coefficients to zero and Ridge regression helps us to reduce multicollinearity, I could not gain a grasp of the effects of these regularization methods on variance and bias.

I am looking for a possible mathematical or intuitive explanation of how variable elimination affects model variance and bias.

Answer 1

They both use regularisation, which tends to shrink the regression coefficients towards zero, and in doing so introduce bias but can reduce variance. They can handle multicollinearity and even having more predictors than observations.

The hope is that they can reduce the mean square error (sum of variance and the square of bias) on out-of-sample predictions. Their effects can be adjusted through tuning; cross-validation can be used as a tool to tune and even to choose between them or to choose some intermediate method such as elastic net.

As you say, the biggest noticeable difference between them is that Lasso often reduces some of the regression coefficients to $0$ in situations where Ridge regression does not. Which works better in different situations is often an empirical rather than a theoretical question.