5
$\begingroup$

My confusion stems from reading the following paper

http://www.bmj.com/content/351/bmj.h3868

It states in its abstract (and they later show an empirical study that conforms to the claim) - "Overfitted models tend to underestimate the probability of an event in low risk patients and overestimate it in high risk patients"

I'm confused with regards to the generality of this statement. I've seen many cartoons/figures depicting overfitting as a model modelling/capturing noise, and it isn't intuitively obvious to me how that noise would necessarily lead to overestimates of risk in high risk and underestimates of risk in low risk patients. Why can't an overfitted model capture noise in a way that it underestimates the risk in high risk patients? Is there a mathematic proof for their statement?

For the second part of my question, I would like to ask, if the claim they make is true, would calibration plots of underfitted models go the other way (slope<1 for observed v/s predicted; underestimating high risk risks, and overestimating low risk risks?). Again I can't intuitively predict why a simpler underfitting model would necessarily generate a predictable calibration curve.

$\endgroup$
  • 1
    $\begingroup$ The statement seems dubious to me. What about models that are approximately unbiased but just have very high variance? $\endgroup$ – Jake Westfall Aug 26 '17 at 17:09
  • $\begingroup$ The statement sounds right to me. That's what overfitting is in a binary context. Consider that you have maximum uncertainty when your guess is .5, & maximum certainty if your guess is 1.0 or 0.0. The ideal fit would be when your guess is equal to the true probability. So overfitting is guesses closer to 0 for observed 0's where the true probability is not 0, and closer to 1 for observed 1's where the true probability is not 1. Conversely, underfitting is guesses closer to .5. $\endgroup$ – gung - Reinstate Monica Mar 28 '18 at 14:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.