Does the best model necessarily have the best results on validation set?

Question

During the fine-tunning of a DistilBert model, I tried two optimizers (with different parameter sets) on the same dataset.

Here are the results:

- AdamW: train loss (0.21), val loss (0.33), accuracy (0.88)
- SGD: train loss (0.35), val loss (0.35), accuracy (0.87)

I read that if:

- train loss > val loss: the model is under-fitted
- train loss == val loss: the model is well-fitted
- train loss < val loss: the model is over-fitted

So I would say that the model trained with AdamW is over-fitted, but on the other end it is (slightly) better.

Should I prefer a well-fitted model with a slight loss of validation results, or should I focus only on the best validation results?

Answer 1

A training loss which is lower than the validation loss doesn't necessarily indicate an overfitted model. Overfitting should be a concern when the validation loss stops improving or actually deteriorates with further model training. There are a number of reasons why training loss might be persistently lower than validation loss and in fact this is typically the case.

Answer 2

First of all, what you are describing are not KPIs. KPIs are usually business metrics for the problem you are trying to solve. They do not have to do anything with the machine learning metrics. Using those terms interchangeably would be confusing for many people. If indeed you had a key performance indicator, it would be a key metric for your project, so it would be decisive by itself.

Second, your definitions of underfitting and overfitting are not correct. Metrics are real-valued, so there is literally zero probability that training and test metrics would be equal. As @Estacionario noticed in their answer, test metrics would usually be worse than the training metrics because they are calculated on unseen data. We are talking about underfitting or overfitting if those differences are significant (there is no formal threshold) and/or based on other criteria.

Finally, consider a more extreme case, where you have two models: the first one has 50% train and 50% test accuracy, while the other has 90% train and 80% test accuracy. Which would you choose? The consistently poor one does not sound like a great choice.