I did a regression on a train data set with 7000 observations and 50 explenatory variables with ols ridge and lasso. The lambda was chosen via cross validation. After that i wanted to compare the prediction accuracy of the 3 models by predict values of a test data set.

I thought i would get a better prediction with lasso and ridge but thats not the case.

How this could be ?What could be the possible reasons?

At the beginning i computed the vif for the ols model and had 3 variables where the vif was over 15. I thought when i have multicollinarity ridge and lasso will allways perform better ?

  • $\begingroup$ What range of possible values was your lambda selected from? And what was the value that was selected? $\endgroup$ Jul 16, 2018 at 11:23
  • 2
    $\begingroup$ It seems something is going wrong with your cross-validation technique. One thing you could do is to set the regularisation parameter $ \lambda = 0 $ for both, because that effectively changes them to an OLS problem. If the CV accuracy is still different, there is an implementational issue. $\endgroup$
    – boomkin
    Jul 16, 2018 at 13:01
  • $\begingroup$ The Lasso sometimes has problems with highly correlated regressors. Consider the Elastic Net. Did you assess your accuracy using MSE? $\endgroup$ Jul 16, 2018 at 13:06
  • $\begingroup$ lambda goes from 10^5 to 10^-3 with the length 100. I use the mse als accuracy measure. I thought Lasso solve the problem of correlated regressor..... $\endgroup$
    – Dima Ku
    Jul 16, 2018 at 14:57
  • 1
    $\begingroup$ Do note that while Lasso and Ridge regression tends to have lower variance than OLS regression, the cost is that they give biased estimates. Remembering that MSE = Bias^2 + Variance, we can note that which method is better depends on the size of the bias and variance. It's not a given that Lasso and Ridge has lower MSE than OLS. $\endgroup$
    – Phil
    Jun 20, 2019 at 11:58


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.