1
$\begingroup$

I am used to associating high variance with overfitting - I don't know how else to think of the physical manifestation of variance wrt an algorithm.

Also, the only cause of overfitting seems to be high variance.

Is there a causal relation between both of these, do they mean the same thing or am I missing something?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I think most statisticians would agree that overfitting isn't necessarily due to just high variance, but rather typically from having a model that is too flexible and not having enough data to constrain the flexibility after seeing the data.

While it is often described as the model being so flexible it just fits the noise, hence I believe why you may think overfitting is just a function of high variance, I can provide a trivial example with no noise that still can suffer from overfitting.

Suppose you have a deterministic linear relation $y = x + 1$. If you have only two points, and you foolishly decide to fit a quadratic equation to those two points, any quadratic function that passes between those two observed points will perfectly fit the data, yet will almost surely make incorrect inference at any two points besides the two observed. Note that the conditional variance in the data in this case is 0!

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks! Do you know of a comprehensive list of causes that exist for variance? $\endgroup$
    – rahs
    Dec 6, 2018 at 1:12
  • $\begingroup$ I don't really follow this answer. Doesn't your model in your example have extremely large variance because there are effectively an infinite amount of quadratics that can fit those two points? When I see the discussion of overfitting, this relates to the variance of the model, i.e. the variance of the estimator for f(X), not the observational noise. $\endgroup$
    – aranglol
    Dec 17, 2021 at 19:56
  • $\begingroup$ @aranglol: you could say I'm being pedantic, but in my example, the variance of the estimator is not fitting noise, but rather from fitting nothing. $\endgroup$
    – Cliff AB
    Dec 18, 2021 at 2:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.