Why do we use Ridge regression instead of Least squares in Multicollinearity?

Which one is correct:

a. lower bias and higher variance
b. lower bias with the same variance
c. higher bias with a lower variance
d. the same bias with lower variance

  • $\begingroup$ Which is correct: I should eat an apple, or I should eat an orange? They are both correct, for different situations. Sometimes you don't mind a bit of bias, sometimes you do. $\endgroup$ – Jeremy Miles Jan 29 '17 at 19:36
  • 5
    $\begingroup$ Is this an exercise/exam question? If so, see stats.stackexchange.com/tags/self-study/info $\endgroup$ – Juho Kokkala Jan 29 '17 at 19:42
  • 1
    $\begingroup$ @JeremyMiles, in this situation the answers are not equally correct, because the situation is well defined: multicollinearity. $\endgroup$ – Richard Hardy Jan 29 '17 at 20:00
  • $\begingroup$ Oh, thanks. You're right, I should have read the question more closely. $\endgroup$ – Jeremy Miles Jan 29 '17 at 20:06
  • $\begingroup$ You do not need to put [solved] in the title. There is an accepted answer, it means that this is solved. $\endgroup$ – amoeba says Reinstate Monica Jan 30 '17 at 11:25

Why would we use ridge regression? It's useful because it prevents overfit to your training data. Now, ask yourself how does preventing overfit influence bias and variance.

I have a useful link for you: When to use regularization methods for regression?

enter image description here

  • $\begingroup$ ok got it! It's higher bias with a lower variance ! Right @student T? $\endgroup$ – Mike Jan 30 '17 at 11:06
  • $\begingroup$ @Mike Yes. You learn. $\endgroup$ – SmallChess Jan 30 '17 at 11:10
  • $\begingroup$ @Mike It's time for you to appreciate our efforts. Please accept an answer, myself or Taylor. This is an opportunity for you to appreciate our efforts. You should see a green tick near our answers. Please pick one. $\endgroup$ – SmallChess Jan 30 '17 at 11:17

Your OLS estimator is $$ \hat{\beta}_{ols} = (X'X)^{-1}X'y, $$ while your ridge regression estimator is $$ \hat{\beta}_{ridge} = (X'X + \lambda I)^{-1}X'y. $$ Take the expectation and variance of each one, and then compare your results.

  • $\begingroup$ ok got it! It's higher bias with a lower variance ! Right @Taylor? $\endgroup$ – Mike Jan 30 '17 at 11:08
  • $\begingroup$ @Mike that's right $\endgroup$ – Taylor Jan 30 '17 at 14:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.