3
$\begingroup$

The following animated diagram shows the training statistics of a Deep Neural Network classifier at the end of each epoch:

Training diagram

The diagrams on the left show the accuracy (upper) and loss (lower) values on training and validation data per epoch. The diagrams on the right show the distribution of confidence values (i.e. maximum of softmax scores) in training (upper) and validation (lower) data.

As you can see, both of training and validation accuracy values stop improving after a certain epoch (i.e. epoch #25) and reach a plateau. However, the confidence of (correct) predictions keeps increasing as we continue the training, while the training and validation loss value still has a decreasing trend (which is also consistent with that). Now:

  • Is it safe to claim that the model's predictions are much more confident in, say, epoch #50 compared to those of epoch #25 and therefore it's a better model to use? (My own answer to that is yes, because that effect is also happening with the held-out validation set.)
  • Is this a good approach - i.e. continuing training after reaching the accuracy plateau, while keeping an eye on loss value - to get a model with much more confident predictions, especially in applications where not only the correctness of prediction is important but also the confidence of the predictions is of great importance? Or is there better alternatives to this? (For example, as one point, I can see a trade-off of training time/computational resources vs. higher confidence.)
$\endgroup$
2
  • $\begingroup$ It seems to me that your model is simply classifying correctly because it has been trained - not being trained further. $\endgroup$ – Paze Nov 21 '20 at 23:33
  • $\begingroup$ @Paze That's right (and I didn't claim otherwise); but the point is, when you compare epoch #25 and epoch #50, further training leads to those (correct) predictions to have a higher confidence. $\endgroup$ – today Nov 22 '20 at 9:15
1
+50
$\begingroup$

In some settings, it would be fine to keep training the network, and the validation loss will continue decreasing towards an asymptote. However, boosting the model's confidence is a poor motivation for continuing the training.

First, it is unclear whether your model becomes more calibrated as you keep training. Typically, these deep neural networks just become over-confident without correctly capturing their probability of correct classification. To check this, you have to use a calibration plot or measures such as Expected Calibration Error (ECE).

Secondly, there are ways of adjusting your model's confidence that are both computationally cheaper and more principled. In general, these calibration methods operate by scaling the confidence via the adjustment of few parameters (or even a single parameter) in the readout layer such that some cost function defined on held-out data is minimized. This function can be cross-entropy if your aim is well-calibrated models.

Reference:

Guo, Chuan, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. "On calibration of modern neural networks." arXiv preprint arXiv:1706.04599 (2017).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.