Is there a better way to describe a model's generalization performance than “under” and “overfitting”?

Question

To me, under and overfitting are the two of the most vague concepts in machine learning.

From Google's first link when you look up these definitions.

A model is said to be underfitted if it "performs badly" on the training as well as test set.

And

A model is said to be overfitted if it "performs well" on the training set but "performs poorly" on the test set.

And it is usually follow by either a graph of the training/validation error plot or some curve associated a particular model (model is never specified, hence curve not reproducible).

I don't need to go into the details why "performs badly, well, good", etc. is subjective and leaves a lot of room for guessing. I also don't want to go into detail why deep network tend not to overfit even when you train for a very high number of epochs. Why is this concept so central to machine learning when it is so vague at the same time?

Is there a better metric or descriptor of generalization of a model as of 2020 than "over/underfitting"?

A more radical idea: should we completely abandon this notion because it is vague?

Answer 1

To understand why we use the relative and subjective notions of under- and overfitting, we must remember that, as George Box said, "all models are wrong" (see here for an explanation of this aphorism), but some of them are useful. When confronted to data whose generative model is unknown, we can define a set of plausible and competing models to explain these data. None of these models will be perfect, in the sense that none of them will perfectly correspond to the ground truth (as Norbert Wiener said, "the best material model of a cat is another, or preferably the same, cat"); but you can compare them in terms of their ability to explain the data and their relative simplicity (i.e. their ability to generalize).

Model selection and comparison are subjective (since you have to define a family of plausible models) and relative (since models are not compared to a ground truth, but to the other competing models) by nature. You probably won't be able to fine the true model, but you can find the best model among the family of models you have at hands. For large data sets, you can compare your competing models based on their training and test errors (as shown in the graph in your question). For smaller data sets, you can use model selection criteria such as the Bayesian Information Criterion or the Akaike Information Criterion.