I have a very simply question: Does it make sense to further train an ANN once it reaches an accuracy of 100 % on the training data?
I'm facing a binary classification problem and read this article containing the following, seemingly contradictory, statements:
If your model has 100% training acc that usually means that the gradient won’t change the model since its already at the lowest loss it can (since its optimizing the train error) so it probably means more epochs won’t change your model in a meaningful way.
One of the most commonly used loss functions in deep learning is the cross-entropy loss, which I’ll use to exemplify the above statement. Consider a simple binary classification problem with labels 0 and 1, and say a training point with true label 0 is assigned output probabilities [0.7, 0.3] by the network. So the training error with respect to this point is 0, because the predicted label is 0. However, the “margin” increases if you continue to train further and the new output probabilities become [0.95, 0.05]. Thus, while the training accuracy remains the same, you’re now making predictions with higher confidence.
Having said that, I must conclude with the following — while in theory your generalization may improve on training further after 100% training accuracy, in practice, you’re almost always overfitting by the time you reach 100% training accuracy, and training further is unlikely to help.
Now I'm a bit confused. Is it really a matter of the loss function used whether training beyond 100% makes sense? What if other regularization techniques such as dropout (instead of early stopping) are used? Wouldn't that allow for a better performance on the test data even if training data accuracy already reached 100%?
Thank you very much for any thoughts on this issue.