I try to understand ridge regression. I think the most important point in it is defining lambda value. I researched some R codes but I didn't understand how to define it. In the example here, in the section copied below:

# Selection of constant is at endpoint.  Extend endpoint and try again
 ridgec <- lm.ridge (y ~ x1+x2+x3c, lambda = seq(0, 1, .1))

# Selection of constant is at endpoint.  Extend endpoint and try again
 ridgec <- lm.ridge (y ~ x1+x2+x3c, lambda = seq(0, 10, .01))

# Final model uses lambda=4.
 ridge.final <- lm.ridge (y ~ x1+x2+x3c, lambda = 4)

How did he decide to use lambda value as 4?

  • $\begingroup$ Note that this isn't really about R. $\endgroup$ – gung - Reinstate Monica Apr 21 '16 at 15:24
  • $\begingroup$ I know it isnt about R.I try to understand the logic of defining lambda value in ridge regression $\endgroup$ – hncl Apr 21 '16 at 15:30

Generally this is done using cross validation. I wont go into that here, as there are extensive resources on this site about how to tune $\lambda$ in ridge regression using cross validation.

In your example, this does not seem to be the case, the select function is doing the work. Here's the source for select as applied to ridge regression objects

> getAnywhere(select.ridgelm)
A single object matching ‘select.ridgelm’ was found
It was found in the following places
  registered S3 method for select from namespace MASS
with value

function (obj) 
    cat("modified HKB estimator is", format(obj$kHKB), "\n")
        cat("modified L-W estimator is", format(obj$kLW), "\n")
    GCV <- obj$GCV
        if (length(GCV)) {
            k <- seq_along(GCV)[GCV == min(GCV)]
            cat("smallest value of GCV  at", format(obj$lambda[k]), 

Looks like the select function is using a statistic called GCV to select $\lambda$, by finding the value that minimizes GCV. I believe this is generalized cross validation. I am not an expert in this area (I tend to just use straight cross validation, and I would recommend you do the same), but if you are very interested, here is a paper that seems to have all the details:

| cite | improve this answer | |
  • $\begingroup$ As a tangential comment, I would also recommend using the glmnet library over lm.ridge. It is fast, modern, and has many wonderful features. $\endgroup$ – Matthew Drury Apr 21 '16 at 16:05
  • $\begingroup$ Thank you Matthew.The page is so complex,i will try to understand it $\endgroup$ – hncl Apr 21 '16 at 18:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.