# Strange batch loss in keras

Im training a Bidirectional RNN with keras.losses.MSE and have my dataset shuffled before training. I manually split it into validation and train data. However when i start training loss follows some strange pattern like the one on the picture with batch loss having drops at the epoch end.

Secondly, sometimes loss has this pattern: gradually decreasing batch loss, then huge drop at validation loss at the epoch end, and after this new epoch batch losses start with the last validation loss, but not the last batch loss, just like my NN learned something during validation data testing.

Also, i have some concerns about MSE loss, which increases proportionally to the batch size. It is MEAN square error, so should not it be independent from the batch size?

First question - loss curve:

Usually loss curves are plotted based on epoch-loss, but you are displaying the loss of every single mini-batch and I think (correct me if I'm wrong) that the loss you are showing is not the mean of but the sum of it.

The last mini-batch that passes thru your model is probably very small (it is the remainder of the what is left after dividing the full dataset into batches of equal size), so it results in a smaller summed-loss, if you would display the mean loss, it will stop happening.

• Thanks, very informative! Looks like keras's mse loss indeed calculates sum of losses in batch, so i switched to tf.losses.mean_squared_error, since im using Tensorflow backend – mcstarioni Jan 14 at 9:55