2
$\begingroup$

I'm wondering about techniques like ridge regression with regard to both multicollinearity and outliers.

My understanding is that ridge regression is primarily used for multicollinearity, but that somehow it is robust to outliers.

If you have a multicollinearity problem, what exactly is it that ridge regression gives you? Isn't the solution to remove the problematic variables? Does ridge regression simply make one of your covariates non-significant and signify that you should remove it?

If you have an outlier problem, what does it mean that ridge regression is robust to outliers? I tried it and using ridge regression gave me more outliers in terms of standardized residual diagnostics than OLS.

$\endgroup$
8
  • $\begingroup$ Dropping a feature is one way to address multicollinearity, but why is multicollinearity a problem in the first place? $\endgroup$
    – Dave
    Commented Dec 7, 2021 at 14:07
  • 1
    $\begingroup$ @Dave Just because you could have simply included highly correlated variables not knowing they were correlated? That's why I'm wondering what the advantage of ridge regression is, because if there's multicollinearity should you not be dropping the variable? Even if you get lower coefficients that generalize better to new data, wouldn't the model generalize even better if you just removed correlated variables? $\endgroup$
    – fmtcs
    Commented Dec 7, 2021 at 14:26
  • $\begingroup$ @fmtcs your are totally correct! :) Please have a look at my answer below and give me an upvote if you think I am on the right track! $\endgroup$ Commented Aug 2, 2022 at 16:15
  • $\begingroup$ @JohnSonnino The upside to dropping a variable is reduced variance. The downside is adding bias by forcing a coefficient to be zero. My answer discusses the interplay between these competing notions. Your stance seems to ignore the bias introduced by forcing a parameter estimate to be zero, even though the parameter probably is not zero. If you’ve ever seen how a constant can be an admissible estimator, even if a silly one, the idea here is similar. $\endgroup$
    – Dave
    Commented Aug 2, 2022 at 16:20
  • $\begingroup$ Dave, again you need to research what multicolinearity means $\endgroup$ Commented Aug 2, 2022 at 16:25

1 Answer 1

6
$\begingroup$

I understand the argument about dropping variables when they are correlated. After all, if variables are correlated, the information contained in one variable is partially contained in another. Including fewer variables results in fewer parameters in the model, and this can lead to a reduction in variance. Combined, I understand the appeal of dropping a variable: retain most of the information but drop a parameter.

The trouble is that, while you decrease the variance, you can introduce bias, perhaps enough that you are worse off, despite the decreased variance.

Regularization techniques are an alternative to dropping variables. They introduce bias, yes, but they do so in a way that does not completely drop variables. For instance, LASSO regularization tends to result in many coefficients calculated as $0$. If you consider that a feature selection step$^{\dagger}$ and run your regression on the "surviving" features that have nonzero coefficients, you will get different results, since (among other reasons) the "dead" features still contributed to the coefficient calculation.

Ridge regression does not set coefficients all the way to zero, and ridge regression seems to have a tendency to outperform LASSO.

None of these techniques---ridge, LASSO, and manually dropping variables---are inherently better than each other. In the link, Harrell gives his arguments for ridge regression but concedes that there are situations where LASSO can do better. If you have theoretical knowledge about the process or have a signal in the data screaming at you to drop a variable, perhaps that would work best.

$^{\dagger}$The link also discusses the issues with using LASSO to select features.

$\endgroup$
12
  • $\begingroup$ This answer is unfortunately incorrect. Please make some investigation into the effects of multicolinearity. $\endgroup$ Commented Aug 2, 2022 at 15:56
  • $\begingroup$ One should never sacrifice the reliability of their results for the inclusion of additional variables that contribute to a model that cannot be implemented effectively in real life situations. Moreover it would be impossible to select the final model considering that the TSS will be chock full of overfitting variances. $\endgroup$ Commented Aug 2, 2022 at 16:54
  • $\begingroup$ Unless you’re using a different definition, $TSS=\sum_{i=1}^n\big(y_i-\bar y\big)^2$ and makes no reference to a model, model features, or model predictions (except in a technical sense where $TSS$ can be seen as the performance of an intercept-only model that always predicts $\bar y$). $\endgroup$
    – Dave
    Commented Aug 2, 2022 at 17:11
  • $\begingroup$ The TSS would be the result of a bad selection practice, since ESS = TSS - RSS $\endgroup$ Commented Aug 2, 2022 at 17:20
  • 1
    $\begingroup$ Overfitting is bad and having many features can contribute to overfitting, sure, but discarding features just because they correlate with other features ignores the opportunity to damage your analysis with underfitting. $\endgroup$
    – Dave
    Commented Aug 2, 2022 at 20:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.