During the fine-tunning of a DistilBert model, I tried two optimizers (with different parameter sets) on the same dataset.

Here are the results:

    - AdamW: train loss (0.21), val loss (0.33), accuracy (0.88)
    - SGD: train loss (0.35), val loss (0.35), accuracy (0.87)

I read that if:

    - train loss > val loss: the model is under-fitted
    - train loss == val loss: the model is well-fitted
    - train loss < val loss: the model is over-fitted

So I would say that the model trained with AdamW is over-fitted, but in the other end it is (slightly) better.

Should I prefer a well-fitted model with a slight loss of KPIs, or should I focus only on the KPIs?