Ridge regression — why does the model only care to control large outliers?

One of the purposes of ridge regression is to curb the effects of outliers which may cause the regression coefficients to be so large and hence cause a highly biased model.

That's why the constraint $\Sigma\beta_j^2<s$ is imposed, forcing the coefficients to not exceed a certain value.

Here is my issue. An outlier could be a value that is either too large or too small. I think this should mean than outliers could cause the $\beta_j$'s to be too small or too large. The formulation of this constraint inequality seems to only care about controlling those outliers which might make the $\beta_j$'s too large.

I believe Ordinary Least Squares Regression equally suffers from the effects of very small outliers.

Can someone please explain if or how Ridge regression controls the "small" outliers as well?

• By "small outlier" do you mean "negative number with large absolute value" or "small in absolute value"? – Matthew Drury Jan 17 '16 at 21:44
• Or maybe, more precisely "far away from the mean in the positive direction, or in the negative direction" is what you mean? – Matthew Drury Jan 17 '16 at 21:54
• If the signal values are 1, 2, 4, 2, 2, 1, 3, 5, 100, -200; then 100 and -200 are candidates for being outliers. That's my context of an outlier. – Minaj Jan 17 '16 at 21:57
• its not for outliers at all. you are right that outliers have a disproportionate effect on the regression coefficients. but ridge regression is to prevent large coefficients mainly from collinearity. – seanv507 Jan 17 '16 at 21:58

Ridge regression is a modification of linear regression. Linear regression is the best linear unbiased estimator (BLUE). They key word is "unbiased." Linear regression should give the smallest mean square error (MSE) of any unbiased linear estimator. Ridge regression adds bias in exchange to reduce variance, thus creating a biased estimator with potentially lower MSE. It does this with a regularization or "smoothing" parameter to shrink the coefficients as you describe. This has the effect of decorrellating variables for differering values along the parameter.

Techniques that are based on the squared loss function are particularly sensitive to outliers. To alleviate this, least absolute square error or even least median squares can be used as the loss function. However, ridge regression by itself is not meant to be used to reduce the effect of outliers because it is just a slightly modified version of linear regression estimated with an identical loss function, but penalized to adjust some of the assumptions in the linear regression model. There is nothing here that explicitly deals with outliers.

• @JohnK But my question would still hold , albeit slightly modified. Does multicollinearity only cause regression coefficients to be large? cant it cause them to be so small as well? If so, ridge regression only seems to care about the former problem. – Minaj Jan 18 '16 at 20:10
• The regularization/coefficient path from ridge regression is a continuous path that leads to the least squares coefficients. It becomes similar to a feature reduction technique, reducing or even removing the influence of correlated variables. The maximum value the ridge regression coefficients should take is the least squares estimate. – iuppiter Jan 18 '16 at 20:22
• Multicollinearity can cause one coefficient to be large and the other small. Ridge Regression coefficient path would start with only one of these variables and then as it moves towards the LS coefficients it should add the less relevant correlated variable. – iuppiter Jan 18 '16 at 20:27
• @orbis I would disagree with "Multicollinearity can cause one coefficient to be large and the other small". Its more that multicollinearity can cause a large negative and (an 'offsetting') large positive coefficient [I assume throughout that two or more inputs are positively correlated with each other]. since ridge regression penalises the squared coefficients ridge regression favours smaller same sign weights (ie averaging inputs) – seanv507 Jan 18 '16 at 23:09

The ridge estimator is very susceptible to outliers, much like the OLS estimator. The reason for that is that we still depend on the least squares minimization technique and this does not allow large residuals. Hence the regression line, plane or hyperplane will be drawn towards the outliers.

There has been considerable work in recent years in order to "robustify" ridge regression. In case you want to read further, here is one paper that advocates an approach based on M-estimators.

Maronna, Ricardo A. "Robust ridge regression for high-dimensional data." Technometrics 53.1 (2011): 44-53.