I'm familiar with concepts like early stopping, and detection of plateau and so on.
Tensorflow CNN training has a possibility of saving only best model too, according to model's accuracy metric (for example). However I do not completely trust saving the best model, and let me explain why.
Assuming this picture example of my real experiments with some data (plot for validation set) 
It is often happening that the curve has some 'good accuracy' spike (red arrow) during the training pretty early. It gets back right after and continues training and keeps slowly increasing accuracy.
Imagine if I don't have the resources/operating time and decide to stop training before epoch number 200, for example. In this case red arrow result will be the best. However in general curve keeps improving. Red curve indicates the moving average accuracy. Moreover, if Early Stopping callback is set-up it will most probably halt the process even before epoch 100, because too many epochs before the improvement happens really! And it happens after 200th epoch.
I also am curious about the further generalization of the network, after 'best' red-arrow result. It looks like it is improving at average.
The loss also seem to gradually decrease:
My main question: Is red-arrow result really, (epoch ~75) is better then epoch ~150 result? Because absolute accuracy is higher, but mean is lower.
Additional advices is also appreciated: I generally use only train-validation split, does this indicates that 3rd split (test) should be presented? If I'm low on data how should I deal with it? What is general strategy of analyzing such cases, looking at the mean accuracy, spikes etc.?
P.S some details about the task: it is multi-class classification problem with ~2500 samples, where I use fixed 500 images for validation part. Adam optimizer is used with fixed learning rate
