2
$\begingroup$

I have a dataset which I split as 80% training and %20 validation sets. (38140 images for training, 9520 for validation) Model that I train is a deeper (~45 layers) convolutional neural network.

I got the below results in the first epochs of training:

Epoch 1: train loss: 1041.52 - validation loss: 1045.89
Epoch 2: train loss: 750.78  - validation loss: 749.95
Epoch 3: train loss: 425.88  - validation loss: 423.35
Epoch 4: train loss: 320.29  - validation loss: 319.35
Epoch 5: train loss: 305.41  - validation loss: 305.07

As can be seen, after first epoch the validation error is slightly lower than training loss. Is it something that I worry or Is it an indicator of good convergence and generalization?

$\endgroup$

2 Answers 2

3
$\begingroup$

In your case the difference is tiny (< 1%), I am quite sure, that this is no problem. The train set may contain more difficult images than the test set, therefore giving a higher loss.

I would interpret this example as having a good generalization without overfitting, plus a little random variation between training and test set.

For more possible reasons, you can check this excellent answer.

$\endgroup$
2
  • $\begingroup$ Is 3% lower than training loss acceptable? $\endgroup$
    – June Wang
    Commented Aug 30, 2022 at 11:03
  • $\begingroup$ Can be acceptable, but as I do not know your scenario you will have to decide for yourself. Look if your test dataset is too small, or if the train-test split has some bias. You can just recheck with a different train-test split, if it is always lower than for the training you should have a look into other reasons or problems. $\endgroup$
    – Simon
    Commented Aug 31, 2022 at 8:52
2
$\begingroup$

Keras calculates train loss and validation loss differently. Train loss is the accumulated loss of each batch. On first batch train loss is going to be poor because the model hasn't seen the whole epoch.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.