My training loss goes down and then up again. It is very weird. The cross-validation loss tracks the training loss. What is going on?
I have two stacked LSTMS as follows (on Keras):
model = Sequential() model.add(LSTM(512, return_sequences=True, input_shape=(len(X), len(nd.char_indices)))) model.add(Dropout(0.2)) model.add(LSTM(512, return_sequences=False)) model.add(Dropout(0.2)) model.add(Dense(len(nd.categories))) model.add(Activation('sigmoid')) model.compile(loss='binary_crossentropy', optimizer='adadelta')
I train it for a 100 Epochs:
model.fit(X_train, np.array(y_train), batch_size=1024, nb_epoch=100, validation_split=0.2)
Train on 127803 samples, validate on 31951 samples