A lot of times I see the following behavior of the training and validation loss during training of deep nets:

At the very beginning (first couple of batches), the training loss descends, but the validation loss increases (black line), only to rapidly catch up with the training loss (red line) after roughly the 8th batch.

enter image description here

It looks as if in the early examples, the model "learns" non-generalizable features, and this gets solved after seeing more examples, improving in generalization.

But if the model starts by learning non-generalizable things (overfitting to the first batches), the training loss would also have huge jumps for the following batches, but this is not observed.

What could be the reason for this early validation loss jump?

  • 1
    $\begingroup$ Is the training loss on the batch calculated after or before the batch has been inputted? Also, is x axis the batch number or epoch, or they are the same here? $\endgroup$
    – gunes
    Dec 15, 2021 at 19:46
  • $\begingroup$ @gunes the training loss shows the loss after prediction of each batch, so after the batch as been inputted (how could it be calculated before? Not sure if I understand). X axis is batch numbers. $\endgroup$
    – hirschme
    Dec 15, 2021 at 21:54


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.