4
$\begingroup$

Unsure why I'm consistently seeing a higher training loss than test loss in my model:

from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
from keras.layers import Activation
from keras.layers import Dropout

model = Sequential()
model.add(LSTM(200, return_sequences=True,input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(LSTM(100, return_sequences=True))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(LSTM(50))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(1))

model.compile(loss='mean_squared_error',optimizer='adam')

history = model.fit(train_X,train_y, epochs=50, batch_size=30, validation_data=(test_X, test_y), verbose=2, shuffle=False)

plt.plot(history.history['loss'], label='train')
plt.plot(history.history['val_loss'], label='test')
plt.legend()
plt.show()

The plot this produces: enter image description here

What am I doing wrong?

$\endgroup$
3
$\begingroup$

You may not be doing anything wrong, this happens sufficiently frequently that is covered in the Keras FAQ. The FAQ give at two common possibilities:

  1. Dropout (and other types of regularisation) are turned off during testing, or
  2. Training loss is calculated over the whole epoch while test loss is calculated only at the end.

If neither of these explain your situation, there are some tips for debugging neural networks in this Github issue.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.