Is it possible to over-train a classifier?

Question

Context: I'm constructing a CNN classifier for text categorization. I have a dataset with 20 different classes and approximately 20,000 labeled features (the 20 News Group dataset for those interested).

I'm wondering if I'm training my model on too many epochs, which would make it really good at categorizing the features from my training dataset, but unable to adapt to new / slightly different inputs. Is that what we call "overfitting"? The term is not clear to me.

Also I would like to clarify the term "convergence" of a neural network. Is this convergence attained when the accuracy starts plateauing? Or is it related to the loss value?

Answer 1

Pankaj Daga's expansion is great, I'll take care of the illustration. Here is a typical curve when training a neural networks:

The reported F1-score for the test set should to be the F1-score of the test set of the epoch where the F1-score of the validation set was the highest. (this is called "test best" on the figure)

Answer 2

Your comment regarding the epoch is true. So, if you use too few epoches, you may underfit and using too many epoches can result in overfitting. As you know you can always increase the training accuracy arbitrarily by increasing model complexity and increasing the number of epoch steps. One way to try and alleviate this problem could be through early stopping. In pseudocode:

• Split data into training, validation and test sets.
• At every epoch or every N epoch:
  • evaluate network error on validation dataset.
  • if the validation error is lower than previous best, save network to epoch.
• The final model is the one with the best performance on validation set.

This is very similar to the classical cross validation techniques you use in machine learning approaches.

Regarding convergence, you usually say the network has converged to some local minima if your error metric and weights are relatively constant over several iterations.