1
$\begingroup$

I saw quite a few discussions related to the problem of high training accuracy with low validation accuracy and what steps to take to address it. I have the same problem with a binary classification case. However, I just want to know if the fact that the neural network reaches high training accuracy (25-30% higher than the validation set) means that it has the potential to be improved for the validation set too?

Is there a possibility that the training accuracy would remain high in any case but I have nothing to do to improve the validation accuracy?

$\endgroup$
5
$\begingroup$

Does high training accuracy for a NN mean that it has a potential to reach high validation accuracy?

No.

Neural networks tend to overfit easily. They can even fit a dataset with random labels perfectly [1]. In that case the training error is zero, but no model can perform better than random guessing on the test set. So, in general case, high training accuracy does not tell you anything about the test accuracy or potential to reach good test accuracy.

See some threads related to this problem:


[1]: Zhang, C., Bengio, S., Hardt, M., Recht, B. and Vinyals, O., 2016. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.