During the fine-tunning of a DistilBert model, I tried two optimizers (with different parameter sets) on the same dataset.

Here are the results:

- AdamW: train loss (0.21), val loss (0.33), accuracy (0.88)
- SGD: train loss (0.35), val loss (0.35), accuracy (0.87)

I read that if:

- train loss > val loss: the model is under-fitted
- train loss == val loss: the model is well-fitted
- train loss < val loss: the model is over-fitted

So I would say that the model trained with AdamW is over-fitted, but in the other end it is (slightly) better.

Should I prefer a well-fitted model with a slight loss of KPIs, or should I focus only on the KPIs?