4
$\begingroup$

I think bias and variance are metrics to choose the moderate model complexity(to choose the right model from candidate models), and underfitting and overfitting are metrics to know to what extent has the model leant the data. Am I right?

And what is the relationship between them?

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
5
$\begingroup$

All are metrics to find the best model: You would like to have an unbiased minimum variance estimator related to the validation interval - and thus neither over nor underfitted. But how to balance these metrics is up to your application / context.

Bias: The error you have for certain even if you can use an infinite number of cases / records. Usually you try to get unbiased models.

Variance / Significance: Relates to the probability that the true relationships between your variables are trivial (e.g. zero) and your model sees an accidental and purely randomly generated data pattern. Variance and bias are usually independent.

Overfitting: Is related to the variance, but it's not the same. If you have a large data matrix you may fit a model with a large number of covariats and many parameters may have a small variance. Nevertheless if you split off 20% of the training material and make a prediction with your model on that validation data, the model may predict worse. That's overfitting: Your model fitted relationships, which aren't randomly within your full data set, but aren't systematic and stable for extrapolations outside the training data set.

Underfitting goes usually with a biased distribution of your residuals: Your model is insufficiently specified. (Relationship between target and covariat is quadratic, but you fit linear). Underfitting can result in a bias, but doesn't have to do. In any case it results in a non-minimal variance. Hope that helps a little bit.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Nice summary but something got lost in it for me. Namely, the very important point that bias-variance tradeoff is a theoretical construct which is very difficult to translate empirically. $\endgroup$
    – Mike Hunter
    Feb 23, 2017 at 14:48
  • $\begingroup$ Contrary to your last paragraph, I think underfitting must result in bias. Underfitting means the model is simpler (less flexible) than the DGP, and so it could not approximate DGP perfectly even given infinite data and perfect estimation precision, hence there is bias. $\endgroup$
    – Richard Hardy
    Sep 21, 2020 at 9:41
  • $\begingroup$ Overfitting is not when prediction is less accurate on the test set. You always have lower accuracy on the dev/test set, it doesn't mean you overfit all the time. For what I've seen, overfitting happens when the accuracy on the test set starts to decrease during training on the training set. $\endgroup$
    – Ben Butterworth
    Sep 27, 2020 at 19:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.