My recurrent neural network (LSTM, resp. GRU) behaves in a way I cannot explain. The training starts and it trains well (the results look quite good) when suddenly accuracy drops (and loss rapidly increases) - both training and testing metrics. Sometimes the net just goes crazy and returns random outputs and sometimes (as in the last of three given examples) it starts to return same output to all the inputs.
Do you have any explanation for this behavior? Any opinion is welcome. Please, see the task description and the figures below.
The task: From a word predict its word2vec vector
The input: We have an own word2vec model (normalized) and we feed the network with a word (letter by letter). We pad the words (see the example below).
Example: We have a word football and we want to predict its word2vec vector which is 100 dimensions wide. Then the input is $football$$$$$$$$$$
.
Three examples of the behavior:
Single layer LSTM
model = Sequential([
LSTM(1024, input_shape=encoder.shape, return_sequences=False),
Dense(w2v_size, activation="linear")
])
model.compile(optimizer='adam', loss="mse", metrics=["accuracy"])
Single layer GRU
model = Sequential([
GRU(1024, input_shape=encoder.shape, return_sequences=False),
Dense(w2v_size, activation="linear")
])
model.compile(optimizer='adam', loss="mse", metrics=["accuracy"])
Double layer LSTM
model = Sequential([
LSTM(512, input_shape=encoder.shape, return_sequences=True),
TimeDistributed(Dense(512, activation="sigmoid")),
LSTM(512, return_sequences=False),
Dense(256, activation="tanh"),
Dense(w2v_size, activation="linear")
])
model.compile(optimizer='adam', loss="mse", metrics=["accuracy"])
We have also experienced this kind of behavior in another project before which used similar architecture but its objective and data were different. Thus the reason should not be hidden in the data or in the particular objective but rather in the architecture.
nan
. $\endgroup$nan
parameter wouldn't result in non-nan loss. My guess is that your gradients happens to explode, similar thing happened to me in non-batch normalized networks. $\endgroup$nan
appeared in one of the computations and then it defaulted into another value that caused the network to go crazy. But it's just a wild guess. Thanks for your opinion. $\endgroup$