1
$\begingroup$

I am currently writing my thesis on deep learning models where I train a VGG like model. I trained my model always with Early Stopping function from Keras, where it stopped training after approximately 100 Epochs. My professor asked why I stop after 100 Epochs and that it is very few epochs. He said I should also try 500, 700, 800 Epochs and see if my model is overfitting or does a better job. After trying out to train my model on 800 Epochs, this is what comes out: enter image description here

Looking at Validation Accuracy and Validation Loss values it looks quite good:

Acc: 1.00; Val Acc: 0.9805; Loss: 0.0019; Val Loss 0.00

But the first thing that comes to my mind is: My proffessor always told us that in the real world or a really realistic model should never have more than let's say 94% of accuracy (given it is a little bit more complex task and not just: is this image black or white). Looking at the image I also see there is a lot of noise for Val Loss. Does that mean my model is overfitting or what can I understand from this.

for more information: I used save best model with the parameter Val Loss because its the only parameter that stagnates so much every time. I have 2 classes with around 8000 images. My learning rate is 0.0001, my val split is 0.35 and batch size is 32 (because bigger batch size causes gpu memory error).

$\endgroup$
2
  • 1
    $\begingroup$ Whether that accuracy is high or low depends a lot on how the classes are distributed in your data. If there's a 50/50 split then sure, 94% is good. But if it's 95/5, then always choosing one class can already achieve 95% accuracy, $\endgroup$
    – kutschkem
    Jun 22, 2021 at 6:58
  • $\begingroup$ The data is splitted equally with each class having around 8.000 images. So I think this shouldn't be the problem right? $\endgroup$
    – NECben067
    Jun 30, 2021 at 18:07

3 Answers 3

2
$\begingroup$

I would suggest this:

  • after 200 epochs, lower your learning rate + lower your batch size.

why?

on some epochs you get a low loss, and some with a high loss, this usually mean that there is a problem in the final "fine tuned" convergence.

reference for why learning rate and batch size are connected here.

good luck!

$\endgroup$
2
  • $\begingroup$ What exactly do you mean with "problem in the finale fine tuned convergence"? I must also explain or try to explain why only the validation loss is so unstable while the other metrics as loss, accuracy and val accuracy is stable after 100 epochs. But all I could think of would be somehting like that the amount of data (8.000 images per class) are too less data for such long epochs so it overfits. $\endgroup$
    – NECben067
    Jun 30, 2021 at 18:16
  • $\begingroup$ @NECben067 1. loss here looks less "stable", because loss can be btween 0 and infinity. accuracy can be onlt between 0 and 1. so a loss of 1000 can result in an accuracy of 0.93 . your validation accuracy is a bit noisier, cosidering the scale. 2. when a model overfits, usualy we see a steady increase in validation loss. here the loss is not steady, meaning that it jumps from high loss to low loss. this means that the model is not converging right - instead of finding a local minimum and staying there, the weights are changing and "jumping". 4 $\endgroup$
    – Jonathan
    Jul 1, 2021 at 8:37
1
$\begingroup$

You have likely overfit, if you get a training accuracy of 100% on a largish training dataset and clearly lower on the validation set. When I calculate confidence intervals for binomial proportions assuming a 80:20 training validation split, the two are quite clearly separated. Accuracy can be a noisy metric (and loss, or focal-loss could be less noisy alternatives to look at), but the mismatch is just too striking. I'd say that's due to training for too long, when you're potentially doing nothing useful for the validation performance (at least with your specific settings) any more and continue to overfit the training performance. That does not mean training for longer might not be a good idea, just maybe not with the current configuration.

How bad this overfitting is in terms of the possibly still useful validation set performance (depends on the application whether that performance is useful, or not), is of course debatable, but you can likely reduce the training-validation gap somewhat with various regularizaiton techniques. Ideas are things like more drop-out, more data augmentation, weight decay, different learning rate/momentum schedule, different loss function such as focal loss etc. (it's a lot less clear whether something like stochastic weight averaging would help). Tuning the regularization and other settings optimally using cross-validation on the training data is the way to go. The things others recommended such as e.g. lowering the learning rate based on some criteria (some alternatives could be schedules like flat-cosine or the one-cycle policy that have a declining learning rate schedule towards the end of training) also sound like a good idea.

Another comment: whether 100 or 800 epochs is a lot or not depends on your setting. It seems like a huge number, if you are doing transfer learning and using a pre-trained model from ImageNet (on large Kaggle competitions I'm more used to using single digit or low double-digit epochs in that situation when using modern learning rate schedules). If you are training a neural network from scratch on ImageNet, this would be a really low number to my mind, on ther other hand for MNIST it seems like a huge number. So, what's a low number of epochs really depends.

$\endgroup$
2
  • $\begingroup$ Thanks for the informative answer! The model is still very useful on unseen images. I tested it with several test datasets and it was doing a great job until one of the test datasets had a brigther contrast. There it classified all images the same which was clearly wrong. I will definitely try regularization out. Why do you think the validation accuracy or all other metrics seem to perform good but the val loss is stagnating that much? What does this say and is it correct if I say because of this stagnating val loss I am choosing 80 epochs as my configuration? $\endgroup$
    – NECben067
    Jun 30, 2021 at 18:43
  • $\begingroup$ If you found this answer helpful, then please consider upvoting and/or accepting it. $\endgroup$ Oct 9, 2021 at 14:15
0
$\begingroup$

The validation accuracy is high, the most possible reason is that the distribution for training and validation is extremely similar. Your dataset is probably also very simple to learn for the model.

The first thing to note is: as your model is performing really well on the validation set, it means it's not memorizing, so it can predict from an unseen set given the distribution is very similar.

Now, if you take your model and apply it on a hard test set, or on a set where the distribution is very different, chances are your model will perform poorly. This is the reason we use regularization and don't overtrain a model because then it is too much dependent on the training distribution.

The spike in the validation loss most probably suggests instability in your model, maybe lower your learning rate over time, also plotting the loss against accuracy is not a good idea either (you can see, your accuracy is always less than 1.0 while the loss can go much higher).

$\endgroup$
1
  • $\begingroup$ Yes the dataset might be simple. I have images of integrated circuits and have to classify them as good or bad ICs. The only difference between the two classes are scratches or broken edges on the integrated circuits. I tested the model on several unseen test datasets and it worked really well. But one test dataset had a much more brighter contrast. There it classified all images as bad, which obviously was false. Why do you think the val loss is so unstable while all the other metrics are stable? What does it mean if the val acc is really good but the val loss always stagnates that much? $\endgroup$
    – NECben067
    Jun 30, 2021 at 18:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.