81
$\begingroup$

I found two questions here and here about this issue but there is no obvious answer or explanation yet.I enforce the same problem where the validation error is less than training error in my Convolution Neural Network. What does that mean?

$\endgroup$
  • $\begingroup$ I don't think this question can be answered without knowing the absolute number of training (cv) and test cases as well as the variance observed for MSE for both cross validation and test. $\endgroup$ – cbeleites unhappy with SX Dec 21 '15 at 10:47
  • $\begingroup$ shuffle the data $\endgroup$ – user0 May 4 '17 at 22:16
  • $\begingroup$ What do we infer from this? Yes, its generated from a dense network with dropout and batchnorm layers. ![enter image description here](i.stack.imgur.com/KX1Fz.png) $\endgroup$ – Srinath Jun 18 '19 at 21:20

10 Answers 10

88
$\begingroup$

It is difficult to be certain without knowing your actual methodology (e.g. cross-validation method, performance metric, data splitting method, etc.).

Generally speaking though, training error will almost always underestimate your validation error. However it is possible for the validation error to be less than the training. You can think of it two ways:

  1. Your training set had many 'hard' cases to learn
  2. Your validation set had mostly 'easy' cases to predict

That is why it is important that you really evaluate your model training methodology. If you don't split your data for training properly your results will lead to confusing, if not simply incorrect, conclusions.

I think of model evaluation in four different categories:

  1. Underfitting – Validation and training error high

  2. Overfitting – Validation error is high, training error low

  3. Good fit – Validation error low, slightly higher than the training error

  4. Unknown fit - Validation error low, training error 'high'

I say 'unknown' fit because the result is counter intuitive to how machine learning works. The essence of ML is to predict the unknown. If you are better at predicting the unknown than what you have 'learned', AFAIK the data between training and validation must be different in some way. This could mean you either need to reevaluate your data splitting method, adding more data, or possibly changing your performance metric (are you actually measuring the performance you want?).

EDIT

To address the OP's reference to a previous python lasagne question.

This suggests that you have sufficient data to not require cross-validation and simply have your training, validation, and testing data subsets. Now, if you look at the lasagne tutorial you can see that the same behavior is seen at the top of the page. I would find it hard to believe the authors would post such results if it was strange but instead of just assuming they are correct let's look further. The section of most interest to us here is in the training loop section, just above the bottom you will see how the loss parameters are calculated.

The training loss is calculated over the entire training dataset. Likewise, the validation loss is calculated over the entire validation dataset. The training set is typically at least 4 times as large as the validation (80-20). Given that the error is calculated over all samples, you could expect up to approximately 4X the loss measure of the validation set. You will notice, however, that the training loss and validation loss are approaching one another as training continues. This is intentional as if your training error begins to get lower than your validation error you would be beginning to overfit your model!!!

I hope this clarifies these errors.

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ Nice answer. There is also the possibility that there is a bug in the code which makes it possible that training has not converged to the optimal soluion on the training set. Or,if the training objective is non convex and the training algorith converges to a local minimum that happens to be good for the validation set. $\endgroup$ – Sobi Dec 18 '15 at 14:49
  • $\begingroup$ @cdeterman thanks.I use RMSE as a performance metric. I've divided my data into 20% for test and 80% for training and validation (20% of training data is cross validated to compute the validation error ). Actually, the Validation error is low, slightly lower than the training error. The test error is higher than training and validation errors. We can find a similar case in MNISTdataset for handwriting recognition stats.stackexchange.com/questions/178371/… $\endgroup$ – Bido Dec 18 '15 at 22:34
  • $\begingroup$ @Bido does my most recent edit address you question? $\endgroup$ – cdeterman Dec 28 '15 at 17:06
  • $\begingroup$ @cdeterman Thanks. I've just noticed that you've edited your answer. It is clear and helpful. $\endgroup$ – Bido Dec 29 '15 at 20:50
  • $\begingroup$ Great explanation, if you could add a few graphs - it would be the best one possible $\endgroup$ – Taras Matsyk Feb 28 '19 at 14:09
135
$\begingroup$

One possibility: If you are using dropout regularization layer in your network, it is reasonable that the validation error is smaller than training error. Because usually dropout is activated when training but deactivated when evaluating on the validation set. You get a more smooth (usually means better) function in the latter case.

| cite | improve this answer | |
$\endgroup$
  • 16
    $\begingroup$ What a simple, sensible answer! $\endgroup$ – rajb245 Mar 6 '17 at 15:12
  • 4
    $\begingroup$ Yes this should be marked as correct answer indeed. $\endgroup$ – Simanas Jun 30 '17 at 11:40
  • 2
    $\begingroup$ I removed my dropout layer, but still see the validation loss lower than the training loss initially! (I am not specifying any regularization on the layers, either!) $\endgroup$ – Josiah Yoder Jul 27 '18 at 18:43
  • 1
    $\begingroup$ @MiloMinderbinder That's an interesting observation. No, I haven't figured anything else out on this yet. $\endgroup$ – Josiah Yoder Dec 13 '18 at 21:07
  • 1
    $\begingroup$ If its the dropout that is really the culprit, is the resulting model really viable ? Is it advisable to reduce the dropout such that this phenomenon disappears ? $\endgroup$ – Kanmani Sep 8 '19 at 22:46
26
$\begingroup$

I don't have enough points to comment on @D-K's answer, but this is now answered as a FAQ on Keras' documentation:

"Why is the training loss much higher than the testing loss?

A Keras model has two modes: training and testing. Regularization mechanisms, such as Dropout and L1/L2 weight regularization, are turned off at testing time.

Besides, the training loss is the average of the losses over each batch of training data. Because your model is changing over time, the loss over the first batches of an epoch is generally higher than over the last batches. On the other hand, the testing loss for an epoch is computed using the model as it is at the end of the epoch, resulting in a lower loss."

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ This does not entirely answer the question either. With dropout disabled, I still see the validation loss about half of the training loss for several epochs in a row! $\endgroup$ – Josiah Yoder Jul 27 '18 at 18:54
  • $\begingroup$ Is your training data representative of the dev data? $\endgroup$ – dter Sep 17 '18 at 10:10
  • $\begingroup$ I randomly split the dataset into training and testing. It visually appeared to be a good sample. I was working on a regression problem where the best classifiers were only slightly better than always predicting the mean value. $\endgroup$ – Josiah Yoder Sep 17 '18 at 13:58
  • $\begingroup$ Your answer does not talk about training loss being greater than the validation loss which is the question that was asked. You are more focused on Training loss and test loss $\endgroup$ – enjal May 19 '19 at 22:06
8
$\begingroup$

my 2 cents: I also had the same problem even without having dropout layers. In my case - batch-norm layers were the culprits. When I deleted them - training loss became similar to validation loss. Probably, that happened because during training batch-norm uses mean and variance of the given input batch, which might be different from batch to batch. But during evaluation batch-norm uses running mean and variance, both of which reflect properties of the whole training set much better than mean and variance of a single batch during training. At least, that is how batch-norm is implemented in pytorch

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ Thank @Mans007, this had happened to me and I was using Keras. The batch-norm layers were the cause. $\endgroup$ – Roei Bahumi Apr 21 '19 at 7:32
7
$\begingroup$

Another possibility that combines both the answer of @cdeterman and @D-K in some way is if you're using some data augmentation mechanism. Infact data augmentation is usually done only on training set and not on validation set (as for the dropout regularization), and this may lead to a validation set containing "easier" cases to predict than those in the training set.

| cite | improve this answer | |
$\endgroup$
3
$\begingroup$

At this time, stochastic gradient based methods are almost always the algorithm of choice for deep learning. This means that data comes in as batches, gradients are computed and parameters are updated. This means you can also compute the loss over the data as each batch is selected. Under this framework, there are two ways in how the loss is computed that I can think of which can lead to this phenomenon that the training error is greater than the validation error. Below, I show that Keras does, in fact, appear to compute the in-sample errors in these ways.

1.) Training error is averaged over whole epoch, rather all at once at the end of the epoch, but validation error is only at end of epoch. As we sample our training data to compute gradients, we might as well compute the loss over them as well. But since the validation data is not used during the computation of gradients, we may decide to only compute the loss after the end of the epoch. Under this framework, the validation error has the benefit of being fully updated, while the training error includes error calculations with fewer updates. Of course, asymptotically this effect should generally disappear, since the effect on the validation error of one epoch typically flattens out.

2.) Training error is computed before batch update is done. In a stochastic gradient based method, there's some noise the gradient. While one is climbing a hill, there's a high probability that one is decreasing global loss computed over all training samples. However, when one gets very close to the mode, the update direction will be negative with respect to the samples in your batch. But since we are bouncing around a mode, this means on average we must be choosing a direction that is positive with respect to the samples out of batch. Now, if we are about to update with respect to the samples in a given batch, that means they have been pushed around by potentially many batch updates that they were not included in, by computing their loss before the update, this is when the stochastic methods have pushed the parameters the most in favor of the other samples in your dataset, thus giving us a small upward bias in the expected loss.

Note that while asymptotically, the effect of (1) goes away, (2) does not! Below I show that Keras appears to do both (1) and (2).

(1) Showing that metrics are averaged over each batch in epoch, rather than all at once at the end. Notice the HUGE difference in in-sample accuracy vs val_accuracy favoring val_accuracy at the very first epoch. This is because some of in-sample error computed with very few batch updates.

>>> model.fit(Xtrn, Xtrn, epochs = 3, batch_size = 100, 
...                 validation_data = (Xtst, Xtst))
Train on 46580 samples, validate on 1000 samples
Epoch 1/3
46580/46580 [==============================] - 8s 176us/sample 
- loss: 0.2320 - accuracy: 0.9216 
- val_loss: 0.1581 - val_accuracy: 0.9636
Epoch 2/3
46580/46580 [==============================] - 8s 165us/sample 
- loss: 0.1487 - accuracy: 0.9662 
- val_loss: 0.1545 - val_accuracy: 0.9677
Epoch 3/3
46580/46580 [==============================] - 8s 165us/sample 
- loss: 0.1471 - accuracy: 0.9687 
- val_loss: 0.1424 - val_accuracy: 0.9699
<tensorflow.python.keras.callbacks.History object at 0x17070d080>

(2) Showing error is computed before update for each batch. Note that for epoch 1, when we use batch_size = nRows (i.e., all data in one batch), the in-sample error is about 0.5 (random guessing) for epoch 1, yet the validation error is 0.82. Therefore, the in-sample error was computed before the batch update, while the validation error was computed after the batch update.

>>> model.fit(Xtrn, Xtrn, epochs = 3, batch_size = nRows, 
...                 validation_data = (Xtst, Xtst))
Train on 46580 samples, validate on 1000 samples
Epoch 1/3
46580/46580 [==============================] - 9s 201us/sample 
- loss: 0.7126 - accuracy: 0.5088 
- val_loss: 0.5779 - val_accuracy: 0.8191
Epoch 2/3
46580/46580 [==============================] - 6s 136us/sample 
- loss: 0.5770 - accuracy: 0.8211 
- val_loss: 0.4940 - val_accuracy: 0.8249
Epoch 3/3
46580/46580 [==============================] - 6s 120us/sample 
- loss: 0.4921 - accuracy: 0.8268 
- val_loss: 0.4502 - val_accuracy: 0.8249

Small note about the code above: an auto-encoder was built, hence why the input (Xtrn) is the same as the output (Xtrn).

| cite | improve this answer | |
$\endgroup$
2
$\begingroup$

I got similar results (test loss was significantly lower than training loss). Once I removed the dropout regularization, both the loss became almost equal.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

@cdeterman and @D-K have good explanation. I would like to one more reason - data leakage. Some part of your train-data are "closely related" with the test-data.

Potential example: imagine you have 1000 dogs and 1000 cats with 500 similar pictures per pet (some owners love to take pictures of their pets in very similar positions), say on the background. So if you do random 70/30 split, you'll get data leakage of train data into the test data.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

A lower validation than training error can be caused by fluctuations associated with dropout or else, but if it persists in the long run this may indicate that the training and validation datasets were not actually drawn from the same statistical ensembles. This could happen if your examples come from a series and if you did not properly randomize the training and validation datasets.

| cite | improve this answer | |
$\endgroup$
-1
$\begingroup$

Simply put, if training loss and validation loss are computed correctly, it is impossible for training loss to be higher than validation loss. This is because back-propagation DIRECTLY reduces error computed on the training set and only INDIRECTLY (not even guaranteed!) reduces error computed on the validation set.

There must be some additional factors that are different while training and while validating. Dropout is a good one, but there can be others. Make sure to check the documentation of whatever library that you are using. Models and layers can usually have default settings that we don't commonly pay attention to.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ It is actually not impossible. Your second part of the answer points into a good direction. Please consider pyimagesearch.com/2019/10/14/… $\endgroup$ – Cadoiz Aug 10 at 3:13

Not the answer you're looking for? Browse other questions tagged or ask your own question.