Difference between bias-variance dilemma and overfitting

Question

I'm wondering what difference it makes whether we talk about bias-variance dilemma where fitting a regression line to the given dataset reduces bias and increases variance or whether we talk about overfitting where the model is fitted to the dataset and does not generalize to new datasets.

Essentially overfitting implies the same tradeoff between biasedness and variance doesn't it?

Answer 1

I think they're related, but not identical problems.

For example, its entirely possible to run into the bias-variance dilemma arguably well before a model is "overfit". For example, in survival analysis one might reasonably argue that a constant hazard (exponential distribution) is so unlikely that you should always "pay" for the protection of using a more flexible distribution like the Weibull distribution. There are a number of other examples where one might struggle with lessening the precision of an estimate well before you've really hit "overfit". Especially in fields where you're often working quite close to the null (environmental epidemiology comes to mind).

To me, overfitting also implies a problem with the interpretation of your model. If one has a regression equation like y = b1 + b2*x + b3*(x^2) + b4*(x^3) + b5*(x^4) + b6*(x^5) you have a problem beyond the bias/precision tradeoff, and into how one meaningfully interprets higher order polynomial relationships into the field.

Answer 2

I view "overfitting" as an example of a bias-variance trade-off (I wouldn't call it a "dilemma") that has gone too far toward the variance end.

The usual example I have in mind for the bias-variance trade-off is the case of a linear model in which all covariates have some effect, but where it would best to drop those whose effects are negligible, incurring some bias in order to reduce the variance.

The usual example I have in mind for "overfitting" is to imagine that the truth is contained within the class of models being considered, so that things could be unbiased, but then the class of models is expanded through added parameters, making it overly flexible, so that the fitted model resembles the observed data quite closely but not necessarily the underlying population or process.

I actually quite dislike the term "overfitting". I'd rather say "fitting too complex a model".