The following animated diagram shows the training statistics of a Deep Neural Network classifier at the end of each epoch:
The diagrams on the left show the accuracy (upper) and loss (lower) values on training and validation data per epoch. The diagrams on the right show the distribution of confidence values (i.e. maximum of softmax scores) in training (upper) and validation (lower) data.
As you can see, both of training and validation accuracy values stop improving after a certain epoch (i.e. epoch #25) and reach a plateau. However, the confidence of (correct) predictions keeps increasing as we continue the training, while the training and validation loss value still has a decreasing trend (which is also consistent with that). Now:
- Is it safe to claim that the model's predictions are much more confident in, say, epoch #50 compared to those of epoch #25 and therefore it's a better model to use? (My own answer to that is yes, because that effect is also happening with the held-out validation set.)
- Is this a good approach - i.e. continuing training after reaching the accuracy plateau, while keeping an eye on loss value - to get a model with much more confident predictions, especially in applications where not only the correctness of prediction is important but also the confidence of the predictions is of great importance? Or is there better alternatives to this? (For example, as one point, I can see a trade-off of training time/computational resources vs. higher confidence.)