# Sudden accuracy drop when training LSTM or GRU in Keras

My recurrent neural network (LSTM, resp. GRU) behaves in a way I cannot explain. The training starts and it trains well (the results look quite good) when suddenly accuracy drops (and loss rapidly increases) - both training and testing metrics. Sometimes the net just goes crazy and returns random outputs and sometimes (as in the last of three given examples) it starts to return same output to all the inputs.

Do you have any explanation for this behavior? Any opinion is welcome. Please, see the task description and the figures below.

The task: From a word predict its word2vec vector The input: We have an own word2vec model (normalized) and we feed the network with a word (letter by letter). We pad the words (see the example below). Example: We have a word football and we want to predict its word2vec vector which is 100 dimensions wide. Then the input is $football$\$.

Three examples of the behavior:

Single layer LSTM

model = Sequential([
LSTM(1024, input_shape=encoder.shape, return_sequences=False),
Dense(w2v_size, activation="linear")
])



Single layer GRU

model = Sequential([
GRU(1024, input_shape=encoder.shape, return_sequences=False),
Dense(w2v_size, activation="linear")
])



Double layer LSTM

model = Sequential([
LSTM(512, input_shape=encoder.shape, return_sequences=True),
TimeDistributed(Dense(512, activation="sigmoid")),
LSTM(512, return_sequences=False),
Dense(256, activation="tanh"),
Dense(w2v_size, activation="linear")
])



We have also experienced this kind of behavior in another project before which used similar architecture but its objective and data were different. Thus the reason should not be hidden in the data or in the particular objective but rather in the architecture.

• did you find out what was causing the problem? – Antoine Jan 10 '18 at 9:33
• Unfortunately not really. We changed to a different architecture and then we did not have a chance to get back to this. We have some clues though. Our guess is that something caused one or more of the params to change to nan. – Marek Jan 10 '18 at 9:50
• nan parameter wouldn't result in non-nan loss. My guess is that your gradients happens to explode, similar thing happened to me in non-batch normalized networks. – Lugi Jan 10 '18 at 10:43
• That's also one of the things we tried to examine using TensorBoard but gradient explosion has never been proved in our case. The idea was that nan appeared in one of the computations and then it defaulted into another value that caused the network to go crazy. But it's just a wild guess. Thanks for your opinion. – Marek Jan 10 '18 at 14:11

Here are my suggestion to pinpoint the issue:

1) Look at training learning curve: How is the learning curve on train set? Does it learn the training set? If not, first work on that to make sure you can over fit on the training set.

2) Check your data to make sure there is no NaN in it (training, validation, test)

3) Check the gradients and the weights to make sure there is no NaN.

4) Decrease the learning rate as you train to make sure it's not because of a sudden big update that stuck in a sharp minima.

5) To make sure everything's right, check the predictions of your network so that your network is not making some constant, or repetitive predictions.

6) Check if your data in your batch is balanced with respect to all classes.

7) normalize your data to be zero mean unit variance. Initialize the weights likewise. It will assist the training.

## protected by SycoraxFeb 22 at 16:15

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).