(not a great question title, I know)


I'm training a neural net on a classification problem with Keras and am consistently noticing that during the training process, the test error is lower than the training error. I thought this was odd and might be unique to the data I'm working with. However, Keras' documentation has the following image, which leads me to believe it isn't uncommon:

enter image description here

(the validation error does cross back over the training error towards the end of the training in the image, but imagine it didn't). At first glance, this seem to be contradictory to the core concept of statistical learning, that being that the training error will almost always underestimate the testing error. The below image, taken from 'The Elements of Statistical learning", shows this concept:

enter image description here

At the end of my training process, I do see what I would expect; the training error is lower than the testing error when evaluated on the entire data sets (train and test respectively).


Given the above, I have a few questions:

  1. Is each point in the Keras loss plot the average loss for that epoch, averaged across each mini-batch in that epoch?
  2. I assume that the error in the second plot (ESL plot) is the loss evaluated over the entire data set (train and test, respectively). Is this the correct interpretation?
  3. If the second plot is the loss evaluated over the entire data sets, this leads me to believe that the difference between the intra-training train and test loss is not a good estimator of the post-training difference. Is this true?
  4. If the above is true, why is the intra-training error not a good estimator of the post-training error (evaluated on the entire data sets)?

1 Answer 1

  1. Usually the losses are averaged over the minibatches; check the implementation of the specific loss function you're using in Keras for details. This answer might help.

  2. I think your interpretation is correct. Furthermore, I don't see a reason why this type of plot (train/val error over epochs) would ever be over something that isn't the entire training split.

  3. So by intra-training loss, you mean end-of-epoch loss over the split; by post-training loss, you mean end-of-training loss over the split. So since these losses should decrease over time, there's no reason to believe that your intra-training losses are good estimates of the post-training loss, given that you should (at least on train w/ sufficient batch size) always be asymptotically crawling towards the minimal loss, simply by virtue of the design of SGD/SGD-style iterative optimizers.

I'm also keeping around my old answer to 3, since it seemed to be somewhat helpful.

  • Old answer to 3: I'm unsure what you mean exactly by intra-training train and test loss vs. post-training difference, but I think I have a general idea based off your previous questions -- forgive me if I'm misunderstanding you.

  • I think by intra-training loss, you mean the loss on a particular minibatch. Depending on the batch size, that'll affect how "good" of an estimator it is for the "post-training loss," which I think means the loss at the end of an epoch. This is a simple sample-size vs. variance argument. In practice, based on my experience, you should only have to worry about the end-of-epoch loss, so you don't have to worry about this question. Let me know if I'm interpreting that correctly.

  1. As for why your validation loss is lower than training loss -- this simply happens sometimes. There is no mathematical rule that says that this cannot happen; in fact, the theorems from stat ML (which might not even apply in the case of a DNN) merely provide a worst case bound (i.e. train/empirical error can be no more than O(something) above val/generalization error) given the suitable assumptions. This is probably a sign that your model is generalizing well on this particular dataset.
  • $\begingroup$ 1. Good point - I'm using Binary Crossentropy. The source code shows the reduction method to be 'SUM_OVER_BATCH_SIZE'; the documentation and the Tensorflow source code confirm that this reduction method does reduce the mean error of the batch. In short, good point. $\endgroup$
    – m_squared
    Commented Jun 14, 2020 at 18:27
  • $\begingroup$ 2. That picture is from page 220 of this book, which is discussing the, "as model complexity increases, you'll see a mostly continual decrease in the training error, but at some point the testing error will go back up, indicating your model has overfit.) $\endgroup$
    – m_squared
    Commented Jun 14, 2020 at 18:30
  • $\begingroup$ 3. Yes and no - by intra-training loss, I'm referring to the end of epoch loss (which, I assume, is the average loss of all the mini-batch losses in that epoch - could be wrong there but that seems intuitive to me). By post-training loss, I'm referring to the loss over the entire training or test set. To your point though, the batch size argument may fit here - currently using 16. It would make sense that the larger the batch size, the better that would be an estimate of the end-of-epoch loss. $\endgroup$
    – m_squared
    Commented Jun 14, 2020 at 18:39
  • $\begingroup$ 4. Despite my previous comments, I think my 4th question still holds some water - which I'll restate to make a little more clear: Why is the end-of-epoch validation loss lower than the end-of-epoch training loss, given that after training is completely done, the training loss (evaluated on the entire training set) is lower that the validation loss (evaluated on the entire validation set)? $\endgroup$
    – m_squared
    Commented Jun 14, 2020 at 18:50
  • 1
    $\begingroup$ I see what you mean by 3 now. Given that the end of epoch loss should ideally decrease every epoch, there's no reason to think of it as an unbiased estimator for the post-training loss (i.e. end-of-epoch loss at the final epoch). For 4, it is certainly possible for the validation loss to be lower than training loss, though that doesn't happen too often. In any case, it's probably a sign that the model is generalizing well to the val set. I'll edit my answer to address these comments. $\endgroup$ Commented Jun 14, 2020 at 18:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.