I am new to machine learning. In an online course that I am taking the instructor claims the following: in linear regression, MLE framework gives us the squared loss cost function which overfits. In order to overcome that we use MAP estimate of the weights. I would like to get a theoretical reason behind why MLE overfits(if it indeed does so) and why using MAP we are able to overcome that(if we can indeed do so).

  • 4
    $\begingroup$ Why do you think that either of your statements - the one about MLE overfitting and MAP not overfitting in linear regression - is true? Note that if you use uniform / diffuse priors, the two will coincide... $\endgroup$ – jbowman Mar 27 '18 at 19:26
  • $\begingroup$ I do not if they are true or not. I saw it here : youtube.com/… $\endgroup$ – Abhay Gupta Mar 28 '18 at 16:36
  • $\begingroup$ @AbhayGupta neither MLE does overfit in general, nor MAP solves any problem with overfitting... $\endgroup$ – Tim Mar 28 '18 at 20:30

Overfitting is usually related to the number of degrees of freedom of the model vs. the data.
It is not related to the method to estimate / calculate the parameters related to those degrees of freedom.

Hence either MAP or MLE doesn't create Overfitting in case the model is correctly chosen.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.