"Overfit" is a commonly discussed concept in ML community. However, I tend to feel that there might be abuse of using this terminology. I wonder what it means when we talk about overfit, especially when we are talking about "overfit on some data".
Specifically, I learned that the text-book definition (from Elements of Statistical Learning or Wikipedia) of overfit is that the test accuracy may drop as model complexity increases. However, I noticed many people also say "a model that overfits to some data". Here are some examples that I have seen/heard:
In a decision tree, if a feature is duplicated, then the tree may "overfit to the duplicated feature" so that the model performance degrades. (I heard this from daily conversation with my peers.)
In a linear model, if pure noise is added to the feature, then the model may "overfit to the noise" so that the model performance degrades. (I heard this from daily conversation with my peers.)
When training a neural network, more epochs will tend to make the network "overfit" because the model starts learning the noise instead of data pattern. (This is a commonly accepted conceptual explanation. For example, see this thread from StackExchange)
For sure, the model performance on test dataset will degrade in all above 3 scenarios. However, is the explanation referring "overfit" a correct one? What exactly does it imply when we say a model is "overfitting on some data"?