14
$\begingroup$

When running ridge regression, how do you interpret coefficients that end up larger than their corresponding coefficients under least squares (for certain values of $\lambda$)? Isn't ridge regression supposed to monotonically shrink coefficients?

On a related note, how does one interpret a coefficient whose sign changes during ridge regression (i.e., the ridge trace crosses from negative to positive on a ridge trace plot)?

$\endgroup$
  • $\begingroup$ Ridge regression only monotonically shrinks coefficients in the case of an orthogonal design matrix. In the presence of correlations, its impossible to say anything of that generality. $\endgroup$ – Matthew Drury Nov 12 '15 at 14:28
5
$\begingroup$

As $\lambda$ increases from zero the contribution of various coefficients changes to suit the optimization, allowing both value increases and sign changes. Have a look at Ryan Tibshirani's ridge regression charts (PDF) illustrating both of your questions (charts 17, 19).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.