Questions on what it means when we talk about "overfit"

Question

"Overfit" is a commonly discussed concept in ML community. However, I tend to feel that there might be abuse of using this terminology. I wonder what it means when we talk about overfit, especially when we are talking about "overfit on some data".

Specifically, I learned that the text-book definition (from Elements of Statistical Learning or Wikipedia) of overfit is that the test accuracy may drop as model complexity increases. However, I noticed many people also say "a model that overfits to some data". Here are some examples that I have seen/heard:

1. In a decision tree, if a feature is duplicated, then the tree may "overfit to the duplicated feature" so that the model performance degrades. (I heard this from daily conversation with my peers.)

2. In a linear model, if pure noise is added to the feature, then the model may "overfit to the noise" so that the model performance degrades. (I heard this from daily conversation with my peers.)

3. When training a neural network, more epochs will tend to make the network "overfit" because the model starts learning the noise instead of data pattern. (This is a commonly accepted conceptual explanation. For example, see this thread from StackExchange)

For sure, the model performance on test dataset will degrade in all above 3 scenarios. However, is the explanation referring "overfit" a correct one? What exactly does it imply when we say a model is "overfitting on some data"?

Answer 1

In statistical learning theory, a quantity of great interest is the gap between empirical risk (expected training error) and population risk (expected test error) when train and test sets are sampled IID from the same population.

In this context, overfitting refers to the size of the gap. No gap means there's no overfitting, large gap = method is overfitting. This is a property of the method rather than model, and refers to expected size of the gap averaged over all train/test pairs, rather than for a specific train/test dataset.

However, you can use gap between specific train/test pair to infer the expected size of the gap. IE, if you see a large difference in train/test errors, then probably the average over all train/test pairs is also large.

No overfitting + low train error also implies low test error.