Validation Error less than training error?

Question

I found two questions here and here about this issue but there is no obvious answer or explanation yet.I enforce the same problem where the validation error is less than training error in my Convolution Neural Network. What does that mean?

Answer 1

It is difficult to be certain without knowing your actual methodology (e.g. cross-validation method, performance metric, data splitting method, etc.).

Generally speaking though, training error will almost always underestimate your validation error. However it is possible for the validation error to be less than the training. You can think of it two ways:

1. Your training set had many 'hard' cases to learn
2. Your validation set had mostly 'easy' cases to predict

That is why it is important that you really evaluate your model training methodology. If you don't split your data for training properly your results will lead to confusing, if not simply incorrect, conclusions.

I think of model evaluation in four different categories:

1. Underfitting – Validation and training error high

2. Overfitting – Validation error is high, training error low

3. Good fit – Validation error low, slightly higher than the training error

4. Unknown fit - Validation error low, training error 'high'

I say 'unknown' fit because the result is counter intuitive to how machine learning works. The essence of ML is to predict the unknown. If you are better at predicting the unknown than what you have 'learned', AFAIK the data between training and validation must be different in some way. This could mean you either need to reevaluate your data splitting method, adding more data, or possibly changing your performance metric (are you actually measuring the performance you want?).

EDIT

To address the OP's reference to a previous python lasagne question.

This suggests that you have sufficient data to not require cross-validation and simply have your training, validation, and testing data subsets. Now, if you look at the lasagne tutorial you can see that the same behavior is seen at the top of the page. I would find it hard to believe the authors would post such results if it was strange but instead of just assuming they are correct let's look further. The section of most interest to us here is in the training loop section, just above the bottom you will see how the loss parameters are calculated.

The training loss is calculated over the entire training dataset. Likewise, the validation loss is calculated over the entire validation dataset. The training set is typically at least 4 times as large as the validation (80-20). Given that the error is calculated over all samples, you could expect up to approximately 4X the loss measure of the validation set. You will notice, however, that the training loss and validation loss are approaching one another as training continues. This is intentional as if your training error begins to get lower than your validation error you would be beginning to overfit your model!!!

I hope this clarifies these errors.

Answer 2

One possibility: If you are using dropout regularization layer in your network, it is reasonable that the validation error is smaller than training error. Because usually dropout is activated when training but deactivated when evaluating on the validation set. You get a more smooth (usually means better) function in the latter case.

Answer 3

I don't have enough points to comment on @D-K's answer, but this is now answered as a FAQ on Keras' documentation:

Why is my training loss much higher than my testing loss?

A Keras model has two modes: training and testing. Regularization mechanisms, such as Dropout and L1/L2 weight regularization, are turned off at testing time. They are reflected in the training time loss but not in the test time loss.

Besides, the training loss that Keras displays is the average of the losses for each batch of training data, over the current epoch. Because your model is changing over time, the loss over the first batches of an epoch is generally higher than over the last batches. This can bring the epoch-wise average down. On the other hand, the testing loss for an epoch is computed using the model as it is at the end of the epoch, resulting in a lower loss.

Answer 4

my 2 cents: I also had the same problem even without having dropout layers. In my case - batch-norm layers were the culprits. When I deleted them - training loss became similar to validation loss. Probably, that happened because during training batch-norm uses mean and variance of the given input batch, which might be different from batch to batch. But during evaluation batch-norm uses running mean and variance, both of which reflect properties of the whole training set much better than mean and variance of a single batch during training. At least, that is how batch-norm is implemented in pytorch

Answer 5

Another possibility that combines both the answer of @cdeterman and @D-K in some way is if you're using some data augmentation mechanism. Infact data augmentation is usually done only on training set and not on validation set (as for the dropout regularization), and this may lead to a validation set containing "easier" cases to predict than those in the training set.

Answer 6

At this time, stochastic gradient based methods are almost always the algorithm of choice for deep learning. This means that data comes in as batches, gradients are computed and parameters are updated. This means you can also compute the loss over the data as each batch is selected. Under this framework, there are two ways in how the loss is computed that I can think of which can lead to this phenomenon that the training error is greater than the validation error. Below, I show that Keras does, in fact, appear to compute the in-sample errors in these ways.

1.) Training error is averaged over whole epoch, rather all at once at the end of the epoch, but validation error is only at end of epoch. As we sample our training data to compute gradients, we might as well compute the loss over them as well. But since the validation data is not used during the computation of gradients, we may decide to only compute the loss after the end of the epoch. Under this framework, the validation error has the benefit of being fully updated, while the training error includes error calculations with fewer updates. Of course, asymptotically this effect should generally disappear, since the effect on the validation error of one epoch typically flattens out.

2.) Training error is computed before batch update is done. In a stochastic gradient based method, there's some noise the gradient. While one is climbing a hill, there's a high probability that one is decreasing global loss computed over all training samples. However, when one gets very close to the mode, the update direction will be negative with respect to the samples in your batch. But since we are bouncing around a mode, this means on average we must be choosing a direction that is positive with respect to the samples out of batch. Now, if we are about to update with respect to the samples in a given batch, that means they have been pushed around by potentially many batch updates that they were not included in, by computing their loss before the update, this is when the stochastic methods have pushed the parameters the most in favor of the other samples in your dataset, thus giving us a small upward bias in the expected loss.

Note that while asymptotically, the effect of (1) goes away, (2) does not! Below I show that Keras appears to do both (1) and (2).

(1) Showing that metrics are averaged over each batch in epoch, rather than all at once at the end. Notice the HUGE difference in in-sample accuracy vs val_accuracy favoring val_accuracy at the very first epoch. This is because some of in-sample error computed with very few batch updates.

>>> model.fit(Xtrn, Xtrn, epochs = 3, batch_size = 100, 
...                 validation_data = (Xtst, Xtst))
Train on 46580 samples, validate on 1000 samples
Epoch 1/3
46580/46580 [==============================] - 8s 176us/sample 
- loss: 0.2320 - accuracy: 0.9216 
- val_loss: 0.1581 - val_accuracy: 0.9636
Epoch 2/3
46580/46580 [==============================] - 8s 165us/sample 
- loss: 0.1487 - accuracy: 0.9662 
- val_loss: 0.1545 - val_accuracy: 0.9677
Epoch 3/3
46580/46580 [==============================] - 8s 165us/sample 
- loss: 0.1471 - accuracy: 0.9687 
- val_loss: 0.1424 - val_accuracy: 0.9699
<tensorflow.python.keras.callbacks.History object at 0x17070d080>

(2) Showing error is computed before update for each batch. Note that for epoch 1, when we use batch_size = nRows (i.e., all data in one batch), the in-sample error is about 0.5 (random guessing) for epoch 1, yet the validation error is 0.82. Therefore, the in-sample error was computed before the batch update, while the validation error was computed after the batch update.

>>> model.fit(Xtrn, Xtrn, epochs = 3, batch_size = nRows, 
...                 validation_data = (Xtst, Xtst))
Train on 46580 samples, validate on 1000 samples
Epoch 1/3
46580/46580 [==============================] - 9s 201us/sample 
- loss: 0.7126 - accuracy: 0.5088 
- val_loss: 0.5779 - val_accuracy: 0.8191
Epoch 2/3
46580/46580 [==============================] - 6s 136us/sample 
- loss: 0.5770 - accuracy: 0.8211 
- val_loss: 0.4940 - val_accuracy: 0.8249
Epoch 3/3
46580/46580 [==============================] - 6s 120us/sample 
- loss: 0.4921 - accuracy: 0.8268 
- val_loss: 0.4502 - val_accuracy: 0.8249

Small note about the code above: an auto-encoder was built, hence why the input (Xtrn) is the same as the output (Xtrn).

Answer 7

I got similar results (test loss was significantly lower than training loss). Once I removed the dropout regularization, both the loss became almost equal.

Answer 8

@cdeterman and @D-K have good explanation. I would like to one more reason - data leakage. Some part of your train-data are "closely related" with the test-data.

Potential example: imagine you have 1000 dogs and 1000 cats with 500 similar pictures per pet (some owners love to take pictures of their pets in very similar positions), say on the background. So if you do random 70/30 split, you'll get data leakage of train data into the test data.

Answer 9

A lower validation than training error can be caused by fluctuations associated with dropout or else, but if it persists in the long run this may indicate that the training and validation datasets were not actually drawn from the same statistical ensembles. This could happen if your examples come from a series and if you did not properly randomize the training and validation datasets.

Answer 10

Simply put, if training loss and validation loss are computed correctly, it is impossible for training loss to be higher than validation loss. This is because back-propagation DIRECTLY reduces error computed on the training set and only INDIRECTLY (not even guaranteed!) reduces error computed on the validation set.

There must be some additional factors that are different while training and while validating. Dropout is a good one, but there can be others. Make sure to check the documentation of whatever library that you are using. Models and layers can usually have default settings that we don't commonly pay attention to.