Does high training accuracy for an NN mean that it has a potential to reach high validation accuracy?

Question

I saw quite a few discussions related to the problem of high training accuracy with low validation accuracy and what steps to take to address it. I have the same problem with a binary classification case. However, I just want to know if the fact that the neural network reaches high training accuracy (25-30% higher than the validation set) means that it has the potential to be improved for the validation set too?

Is there a possibility that the training accuracy would remain high in any case but I have nothing to do to improve the validation accuracy?

Answer 1

Does high training accuracy for a NN mean that it has a potential to reach high validation accuracy?

No.

Neural networks tend to overfit easily. They can even fit a dataset with random labels perfectly [1]. In that case the training error is zero, but no model can perform better than random guessing on the test set. So, in general case, high training accuracy does not tell you anything about the test accuracy or potential to reach good test accuracy.

See some threads related to this problem:

• How to know that your machine learning problem is hopeless?
• Expected best performance possible on a data set

---

[1]: Zhang, C., Bengio, S., Hardt, M., Recht, B. and Vinyals, O., 2016. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530.