2
$\begingroup$

I am running ridge regression on some data using a series of regularization params from 0.0001 to 1000. I was amazed to see that high values of lambda param 500-1000 is giving better results with cross validation.

Is it possible to get better results on such high lambda values?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Yes, it is possible. The order of lambda value depends on the difference between scales of residuals and coefficients. You have observed high lambda becouse your residuals are large when compared with beta coefficients. To make lambda independent of scales you should standardize your response and covariates.

Improve this answer
$\endgroup$
3
  • $\begingroup$ So do I need to scale them within the range of [0 1] or zero mean unit variance is enough. I tried to standardize the inputs and outputs to have zero mean and unit variance but still it is not good enough. I still get very high value for $\lambda$ $\endgroup$
    – user34790
    Commented Sep 4, 2013 at 17:04
  • $\begingroup$ centering your data and standardizing to unit variance is fine. This is actually how the penalize and glmnet packages in R standardize. Another option is to standardize to unit norm, which is often recommended for Lasso regression (which are similar to ridge regression). $\endgroup$
    – David Marx
    Commented Sep 4, 2013 at 17:17
  • $\begingroup$ Zero mean and unit variance should be preferred. If this still has no effect, then something goes wrong. The covariates may be completely irrelevant, or your cross-validation code contain a bug, or something else. It would be useful to see your beta's, mean squared errors and cross-validation mean squared errors for non-standardized and standardized models. $\endgroup$
    – O_Devinyak
    Commented Sep 4, 2013 at 17:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.