3
$\begingroup$

One of the purposes of ridge regression is to curb the effects of outliers which may cause the regression coefficients to be so large and hence cause a highly biased model.

That's why the constraint $\Sigma\beta_j^2<s$ is imposed, forcing the coefficients to not exceed a certain value.

Here is my issue. An outlier could be a value that is either too large or too small. I think this should mean than outliers could cause the $\beta_j$'s to be too small or too large. The formulation of this constraint inequality seems to only care about controlling those outliers which might make the $\beta_j$'s too large.

I believe Ordinary Least Squares Regression equally suffers from the effects of very small outliers.

Can someone please explain if or how Ridge regression controls the "small" outliers as well?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ By "small outlier" do you mean "negative number with large absolute value" or "small in absolute value"? $\endgroup$
    – Matthew Drury
    Jan 17, 2016 at 21:44
  • 1
    $\begingroup$ Or maybe, more precisely "far away from the mean in the positive direction, or in the negative direction" is what you mean? $\endgroup$
    – Matthew Drury
    Jan 17, 2016 at 21:54
  • $\begingroup$ If the signal values are 1, 2, 4, 2, 2, 1, 3, 5, 100, -200; then 100 and -200 are candidates for being outliers. That's my context of an outlier. $\endgroup$
    – Minaj
    Jan 17, 2016 at 21:57
  • 2
    $\begingroup$ its not for outliers at all. you are right that outliers have a disproportionate effect on the regression coefficients. but ridge regression is to prevent large coefficients mainly from collinearity. $\endgroup$
    – seanv507
    Jan 17, 2016 at 21:58

2 Answers 2

Reset to default
3
$\begingroup$

Ridge regression is a modification of linear regression. Linear regression is the best linear unbiased estimator (BLUE). They key word is "unbiased." Linear regression should give the smallest mean square error (MSE) of any unbiased linear estimator. Ridge regression adds bias in exchange to reduce variance, thus creating a biased estimator with potentially lower MSE. It does this with a regularization or "smoothing" parameter to shrink the coefficients as you describe. This has the effect of decorrellating variables for differering values along the parameter.

Techniques that are based on the squared loss function are particularly sensitive to outliers. To alleviate this, least absolute square error or even least median squares can be used as the loss function. However, ridge regression by itself is not meant to be used to reduce the effect of outliers because it is just a slightly modified version of linear regression estimated with an identical loss function, but penalized to adjust some of the assumptions in the linear regression model. There is nothing here that explicitly deals with outliers.

Improve this answer
$\endgroup$
4
  • $\begingroup$ @JohnK But my question would still hold , albeit slightly modified. Does multicollinearity only cause regression coefficients to be large? cant it cause them to be so small as well? If so, ridge regression only seems to care about the former problem. $\endgroup$
    – Minaj
    Jan 18, 2016 at 20:10
  • $\begingroup$ The regularization/coefficient path from ridge regression is a continuous path that leads to the least squares coefficients. It becomes similar to a feature reduction technique, reducing or even removing the influence of correlated variables. The maximum value the ridge regression coefficients should take is the least squares estimate. $\endgroup$
    – iuppiter
    Jan 18, 2016 at 20:22
  • $\begingroup$ Multicollinearity can cause one coefficient to be large and the other small. Ridge Regression coefficient path would start with only one of these variables and then as it moves towards the LS coefficients it should add the less relevant correlated variable. $\endgroup$
    – iuppiter
    Jan 18, 2016 at 20:27
  • 1
    $\begingroup$ @orbis I would disagree with "Multicollinearity can cause one coefficient to be large and the other small". Its more that multicollinearity can cause a large negative and (an 'offsetting') large positive coefficient [I assume throughout that two or more inputs are positively correlated with each other]. since ridge regression penalises the squared coefficients ridge regression favours smaller same sign weights (ie averaging inputs) $\endgroup$
    – seanv507
    Jan 18, 2016 at 23:09
6
$\begingroup$

The ridge estimator is very susceptible to outliers, much like the OLS estimator. The reason for that is that we still depend on the least squares minimization technique and this does not allow large residuals. Hence the regression line, plane or hyperplane will be drawn towards the outliers.

There has been considerable work in recent years in order to "robustify" ridge regression. In case you want to read further, here is one paper that advocates an approach based on M-estimators.

Maronna, Ricardo A. "Robust ridge regression for high-dimensional data." Technometrics 53.1 (2011): 44-53.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.