Evaluating on training data gives different loss

Question

I'm training a simple neural network using keras with the tensorflow backend. As I don't know much about what I'm doing, I'm exploring things a little bit to try to get the hang of it and to try to figure out what is going on. My model is a GRU based RNN. I'm training it on multi-variate time-series data (4 input time series) and the goal is to try to predict future values of one of the inputs.

After the 20th epoch on the training set, Keras reported this:

Epoch 20/20
394311/394311 [==============================] - 50s - loss: 18.9257

Ok, so my loss is right around 19. I wanted to see how the model would generalize to my test dataset:

model.evaluate(x_test, y_test)  # 11.979977535933825

This felt too good to be true (maybe?). I'm actually not quite sure how to interpret a lower loss on the test data than on the training data ... To try to figure that out, I decided to look at the loss computed on the training set:

model.evaluate(x_train, y_train)  # 16.165901696732373

This is clearly not the same value that was reported near the end of my last training epoch. I'm clearly missing something when it comes to knowing how to interpret the loss and how it's calculated... Any insight into why the loss value might be so much lower on the test data than the training data would be great. Also insight into why the loss at the end of the last epoch is so much different than the loss when I evaluate the model on the training data would also be welcome.

Answer 1

Well ... It looks like this is actually in the Keras FAQ:

A Keras model has two modes: training and testing. Regularization mechanisms, such as Dropout and L1/L2 weight regularization, are turned off at testing time.

Besides, the training loss is the average of the losses over each batch of training data. Because your model is changing over time, the loss over the first batches of an epoch is generally higher than over the last batches. On the other hand, the testing loss for an epoch is computed using the model as it is at the end of the epoch, resulting in a lower loss.

Answer 2

When using model.fit and model.evaluate on different datasets, the result will NEVER be exactly the same. There is a multitude of factors, but usually it's just because X_train and X_test are not exactly alike. So the model inherently will not get the exact same prediction results, thus not the same loss.

As you correctly pointed out, you should check whether fit and evaluate on X_train produce a similar loss. They won't be exactly the same (due to how fit vs evaluate works, where one updates the weights after each batch whereas the evaluate uses the same weights throughout the dataset). But they should be pretty close. In your case, loss of 19 vs 16 seems to be different enough that it's an issue......which brings me to a question no one as answered yet ->

That is the bigger issue with Keras (and apparently with Time Series) seems to be that fit and evaluate on the SAME dataset (i.e "X_train") can produce radically different results. That's certainly not normal. I've documented in more detail here where you can see a difference in train vs evaluate loss by an order of magntitude!:

https://datascience.stackexchange.com/questions/113627/lstm-sequential-val-loss-train-loss-on-same-dataset