"Overfit" is a commonly discussed concept in ML community. However, I tend to feel that there might be abuse of using this terminology. I wonder what it means when we talk about overfit, especially when we are talking about "overfit on some data".

Specifically, I learned that the text-book definition (from Elements of Statistical Learning or Wikipedia) of overfit is that the test accuracy may drop as model complexity increases. However, I noticed many people also say "a model that overfits to some data". Here are some examples that I have seen/heard:

  1. In a decision tree, if a feature is duplicated, then the tree may "overfit to the duplicated feature" so that the model performance degrades. (I heard this from daily conversation with my peers.)

  2. In a linear model, if pure noise is added to the feature, then the model may "overfit to the noise" so that the model performance degrades. (I heard this from daily conversation with my peers.)

  3. When training a neural network, more epochs will tend to make the network "overfit" because the model starts learning the noise instead of data pattern. (This is a commonly accepted conceptual explanation. For example, see this thread from StackExchange)

For sure, the model performance on test dataset will degrade in all above 3 scenarios. However, is the explanation referring "overfit" a correct one? What exactly does it imply when we say a model is "overfitting on some data"?


1 Answer 1


In statistical learning theory, a quantity of great interest is the gap between empirical risk (expected training error) and population risk (expected test error) when train and test sets are sampled IID from the same population.

In this context, overfitting refers to the size of the gap. No gap means there's no overfitting, large gap = method is overfitting. This is a property of the method rather than model, and refers to expected size of the gap averaged over all train/test pairs, rather than for a specific train/test dataset.

However, you can use gap between specific train/test pair to infer the expected size of the gap. IE, if you see a large difference in train/test errors, then probably the average over all train/test pairs is also large.

No overfitting + low train error also implies low test error.

  • $\begingroup$ So, are all of the statements in OP legit? I think the key point to my confusion is that when we say overfitting, the gap can be caused by both model being too complicated or data quality being bad, right? $\endgroup$
    – mw19930312
    Apr 20 at 17:49
  • $\begingroup$ Technically overfitting refers to the size of the gap between train error/test error, but colloquially, it's used to refer to things which imply this gap is large. IE "model is overfitting" = you expect large gap between test/train errors $\endgroup$ Apr 20 at 17:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.