I am working on CNN models which are intended to predict a protein's structure from its amino acid sequence. I have a decently large data set, 750 protein structures containing over 100,000 amino acid residues. I am implementing my CNN's in Keras. When I perform 3X cross-validation to help select architectures and hyperparameters, I have 50K examples in my training folds and 25K samples in my validation folds. I am using the ADAM gradient descent algorithm, with the default settings. My backend is Tensorflow.
How many epochs should I train before giving up? I know, that's not an easy question to answer. The machine learning class that I took recommended that a good criterion for stopping is that the validation loss starts increasing. Defining what constitutes "increasing" can be a little vague. Some CNN architectures that I have tried can give noisy loss values for a couple of epochs. I assess my training and validation losses at the end of each epoch, and my "patience value" for validation loss is equal to the square root of the current epoch. I deliberately go a bit farther when the number of epochs is large, so that I have a decent snapshot of what happens after the minimum I chose. Here's an example from one of my typical runs.
In addition to noticing the validation minimum, I also note that the training loss is still smoothly declining when I decide to stop. Although this particular architecture is not that large (about 7,000 trainable weights), it is apparently capable of doing a better job of memorizing the training set than I allowed it to do.
Now, I have also looked at the Tensorflow Playground web site, and I have seen something quite different. The Playground does not attempt to stop when a model is optimized, it just keeps running. Because the Playground data sets are so small, there is no great cost in time or computation to simply keep plugging away.
In the Two Spirals example, it is easy to choose conditions where test loss values climb higher, for tens of epochs, before they finally reverse. And soon after the reversal, a lower test error value is achieved than would have been achieved by choosing some form of early stopping. What exactly is happening in these examples? I haven't yet found the optimizer in the Playground's source code. If it's ADAM, what kind of momentum values are they using? How is it that the Two Spirals solver always seems to best its older local minimum when it swings back?
For my own work, I need to understand whether my early stopping criteria are too conservative. This could be a problem, because I am currently using close to an hour to perform a 3X cross-validation on a single CNN architecture with just three hidden layers. I believe that there are some optimizations I can make in Keras and/or Tensorflow to increase the throughput. However, optimizations of TF and/or Keras may amount to little if I need to train for many more epochs.
Any advice you may have is appreciated, thanks!