1
$\begingroup$

I have read a lot of advantages of regularisation methods (e.g. ridge, lasso), but in what cases might multiple regression with Ordinary Least Squares be more desirable? For example is there a downside to using ridge regression for n >> p problems (where n is the number of samples and p is the number of features), or when there is little multicollinearity between predictor variables? If not then I'm wondering what OLS is good for (except in cases where there's only one predictor variable of course).

$\endgroup$
3
$\begingroup$

Ridge regression helps multicollinearity and overfitting. But sometimes you just don't care about overfitting or multicollinearity:

  • Your want unbiased coefficients and you don't care about their standard errors because you don't want to do any statistical test
  • Your don't need to do predictive modelling - you don't have a test set anyway
  • You don't have that many predictors

Ridge regression might prevent you understanding the details in your data set.

$\endgroup$
  • 2
    $\begingroup$ Could you give me a (concrete) example of a situation where one wants unbiased coefficients and doesn't care about statistical errors? Thanks! $\endgroup$ – user795305 Oct 20 '17 at 13:43
  • 1
    $\begingroup$ One aspect of multicollinearity is that small changes in the data set can make big changes in the parameter estimates. If you care about the coeffiicents at all then you have to care about collinearity. The only time when you do NOT care about it, as far as I know, is when your only aim is prediction. And the number of predictors isn't relevant in any way that I can see. Even with only 2 predictors, there could be collinearity. $\endgroup$ – Peter Flom Oct 20 '17 at 22:00
2
$\begingroup$

The downside of ridge regression is that the parameter estimates are biased. When the variances of the expectations aren't large, then unbiasedness is a lot to give up.

$\endgroup$
  • $\begingroup$ Can you confirm: when you say the parameters in ridge regression are biased, do you mean they have been shrunk? In ridge regression the parameters are shrunk but their relative importances (e.g. b1=0.02 is double as important as b2=0.01) are still at least given? $\endgroup$ – Oliver Angelil Oct 20 '17 at 19:41
  • 1
    $\begingroup$ I don't think they are necessarily shrunk - they might be increased. And I don't think their relative importance is necessarily the same. $\endgroup$ – Peter Flom Oct 20 '17 at 21:58
  • $\begingroup$ Ok. In my case I'm not interested in making predictions. I'm just interested in identifying the relative importance of each one of my X variables to explain the y variable. There is some multicollinearity between the X variables (VIF between ~3 and ~7). I think ridge would be the way to go? $\endgroup$ – Oliver Angelil Oct 21 '17 at 0:03
  • 3
    $\begingroup$ I'm not sure there is a good answer to this. Ridge results in bias and, unless the bias is uniform in a particular way, the ratio of two coefficients could be affected a lot. When variables are colinear, the question of how important each is doesn't really make sense. You could look at each in a bivarite regression with the Y variable, that might be the best answer. But then you are not controlling for other variables. $\endgroup$ – Peter Flom Oct 21 '17 at 12:39
0
$\begingroup$

I hope this helps :

> x1 <- rnorm(30)
> x2<- rnorm(30, mean = x1, sd=0.01)
> y <- rnorm(30, mean = 5+x1+x2)
> cor(x1,x2)
[1] 0.9999316
> fit <- lm(y~x1+x2)$coeff;fit
(Intercept)          x1          x2 
   5.175611    3.859883   -1.452410 
   > 
   > 
   > library(MASS) 
   > lm.ridge(y~x1+x2,lambda = 1)
           x1       x2 
5.171224 1.187841 1.174328 
   > 
   > 
   > 
   > 
   > x1 <- rnorm(30)
   > x2<- runif(30, min = 0, max = 1)
   > cor(x1,x2)
[1] 0.3348988
   > fit <- lm(y~x1+x2)$coeff;fit
(Intercept)          x1          x2 
  5.0691119   0.2017907  -0.6022160 
> lm.ridge(y~x1+x2,lambda = 1)
               x1         x2 
  5.0507017  0.1928616 -0.5704941  

Also, there is a theorem stated that " There always exists a value $\lambda$ such that $$MSE(\widehat{\beta_{\lambda}}) < MSE(\widehat{\beta}^{OLS})$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.