If a neural network performs very well on training data, does that mean it is overfitting?

Question

In a (including convolutional) neural network, suppose I get a fair accuracy (say, $\approx 80\%-90\%$) on my validation/test data, but extremely good accuracy ($\approx 100\%$) if I run it back on my training data.

Does this necessarily mean that my network is overfitting, and I should apply some techniques (get more data, regularize, etc.) to handle it? In other words, can I conclude that my network is overfitting, or very highly likely overfitting? Are there other reasons why this can happen?

Answer 1

The training error is always going to be somewhat smaller than the validation error (because you're actively trying to minimize the training error) - that in itself is not a sign of overfitting. Overfitting means you decreased the training error at the expense of a higher validation ($\approx$ generalization) error. For iterative optimization procedures, you can diagnose this by plotting the training and validation error/performance over time (i.e. over successive iterations of the optimization algorithm). If, at some point, the validation error stops decreasing or even starts increasing while the training error still decreases, that is a pretty sure sign of overfitting.

For learning problems with analytical solutions (e.g. regression), you can see whether the validation performance improves when you apply regularization. If so, then the non-regularized solution was probably overfitting the training data.

Answer 2

I'd say very highly likely. A training accuracy of 100% is acceptable only if the data isn't noisy (which is definitely not the case for image recognition). If not, you probably are learning unnecessary noise, which is not going to allow the net to generalize well, since it "belongs" only to that particular training set instance.