2
$\begingroup$

I was following the below example from 'StatQuest with Josh Starmer' youtube channel.

The example is pretty simple: red line is the usual 'least squares' (for the red points), and the blue one is ridge regression line (for the red points); where we sacrifice a bit of error in the test data, but it will fit better all data (green +red dots).

I do understand the above, and it makes sense; but what if the all the real data ends up being above the line? Why ridge regression only assumes that all the remaining data could be better fit with a smaller slope and not a larger slope?

enter image description here

Improve this question
$\endgroup$
5
  • 3
    $\begingroup$ Possible duplicate of The proof of shrinking coefficients using ridge regression through "spectral decomposition" $\endgroup$
    – Sycorax ♦
    Commented Apr 13, 2019 at 19:35
  • $\begingroup$ Looks like a mathematical explanation/proof. I understand how ridge works and we are minimizing it, we will never get a greater slope than 'least squares'. My question is why? per my view there is a 50/50 chance that the data points will be either above or below, why we only search 'below'? $\endgroup$
    – Chicago1988
    Commented Apr 13, 2019 at 19:42
  • $\begingroup$ I agree, but, any penalty selected will only make my slope lower (or stay the same), but never larger! Why we assume that a lower slope might fit better? why don't we assume that a larger slope might fit it better? $\endgroup$
    – Chicago1988
    Commented Apr 13, 2019 at 21:46
  • 3
    $\begingroup$ Nobody assumes that: it's a result of the model. In fact, when you introduce more than one explanatory variable, the sizes of the slopes actually can increase with Ridge Regression. $\endgroup$
    – whuber ♦
    Commented Apr 13, 2019 at 22:41
  • 2
    $\begingroup$ First, @whuber 's comment seems like it's close to an answer, that answer being "it doesn't necessarily do that". Second, I'm not sure your understanding of ridge regression is correct. Ridge allows for some bias in the parameter estimates in order to reduce the variance of those estimates. $\endgroup$
    – Peter Flom
    Commented Apr 14, 2019 at 12:24

1 Answer 1

Reset to default
4
$\begingroup$

Sizes of the slopes can actually increase with ridge regression. That is because with multiple predictor variables, reducing the norm of the coefficient vector can sometimes be done more effectively if one or some (but clearly not all) of its components is allowed to increase. With simple linear regression (assuming the intercept is not penalized, as is usual) this cannot occur.

One way of seeing this occur is plotting the coefficient paths when the penalization parameter is increased, and some examples of such plot can be seen in this post: Coefficients paths – comparison of ridge, lasso and elastic net regression

Note that in that plot, you can see one trace first increasing, and then ultimately decreasing. That is quite typical.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.