Bad performance on training set and increasing validation loss: Overfitting or underfitting?

Question

From my understanding of over- and underfitting, the two behaviors are not completely mutually exclusive. Overfitting can pretty much always be achieved if the model has enough capacitance (is able to capture the variance of the data) and is trained for long enough time. I have more issues with the concept of underfitting, meaning that the model has a bias problem and can't capture the complexity/variance of the data for the given task.

When evaluating if a model has an overtraining or undertraining problem, there is one case for which I don't know how to make an assessment: The case were a model starts overfitting very quickly (validation loss stops decreasing and starts increasing) but the training loss is still very high and performance is still very bad.

In this case the model is already capturing information that is not generalizable, but which is also not enough for getting a good fit to the training data. How is one supposed to combat this behavior? Is it still a sign of overfitting, for which we have to regularize and stop the model for learning non-generalizable information?

Answer 1

A model that cannot really capture the structure of the data, i.e. a model that actually underfits, can still memorize something internally, that results in bad generalization. Although it is typically not considered as overfitting since your model is not able to capture the training data well, it actually emulates overfitting by spending its resources to memorize (maybe some portion of) your data. In such a case, you should stop the learning process, e.g. early-stopping for neural networks. But, more importantly, you should either update your model or your training algorithm (e.g. gradient descent method).