# Weird behaviour in toy RNN (Keras, LSTM)

I'm trying to learn more about RNNs and I'm tackling a toy problem. I'm generating some data that has a pattern, two 1s followed by three 0s which keeps repeating infinitely without any noise. So my master data is something like [1 1 0 0 0 1 1 0 0 0 1 1 0 0 0 1 ... ]

Then I slide a window of N timesteps over the data and feed this into an LSTM, asking it to predict the next value. I'm treating this as a binary classification problem.

model = tf.keras.models.Sequential([
tf.keras.layers.LSTM(4, input_shape=(None, 1)),
tf.keras.layers.Dense(1, activation='sigmoid')
])

model.fit(train_gen, epochs=EPOCHS)


All is well, the model reaches 100% accuracy and near 0 loss pretty quickly. However, unexpected things start happening when I feed in sequences of different lengths (drawn from the same master data).

for i in range(15):
TEST_WINDOW_SIZE = WINDOW_SIZE + i

longer_data_gen = TimeseriesGenerator(train_data[:2000], train_data[:2000], TEST_WINDOW_SIZE, batch_size=2000)

[loss, acc] = model.evaluate(longer_data_gen)
if acc < 1.0:
print('For i={} accuracy is {}'.format(i, acc))


The output will be something like

i = 0, acc = 1.0
i = 1, acc = 0.6
i = 2, acc = 0.2
i = 3, acc = 0.2
i = 4, acc = 0.6
i = 5, acc = 1.0
...


So basically the network learned the pattern, but it isn't syncing it to the input, it's off-phase.

Note:

• In my experiments, adding dropout=0.15 to the LSTM sometimes fixes the problem, depending on the run, however the reported accuracy on the training set doesn't get to 100%, despite me getting 100% accuracy on all my variable length test data.

• I also tried lowering the number of hidden units in the LSTM but it doesn't seem to do the job

• Sometimes it generalizes even without dropout, but most of the time it doesn't

I kind of get the feeling I'm doing something wrong here, it seems like it's pretty hard to generalize on such a simple problem. Am I approaching this wrong ?

The full code is here.

It seems LSTM hasn't really learnt the short term pattern, which is the only pattern. If you see carefully, it only needs last 3 inputs to "guess", or in your case "predict" (because you used sigmoid) the next number. For instance

i/p    o/p
11000 - 1
01100 - 0
00110 - 0
00011 - 0
10001 - 1


But your LSTM is relying on values that are even further from last 3 digits to predict the next digit. A simple fix of

TEST_WINDOW_SIZE = WINDOW_SIZE*10 + i


should give you an accuracy of 1, irrespective of value of i

Also, the model is overfit. If you get an accuracy of 0.9<acc<1.0 by 5th iteration, then probably the fix won't be needed

• I don't quite understand. training_example_length == WINDOW_SIZE. Unless you mean pattern length ? My pattern length is 5 and WINDOW_SIZE=10. How is WINDOW_SIZE much larger ? In practice, if I know the length of my pattern then what use is an RNN to me? The network should learn the pattern and I feel like I give it enough context for it to be able to. – Adrian Buzea Jun 19 at 16:03
• yes, oversight from me, I thought training_example_length = 32. That said, TEST_WINDOW_SIZE = WINDOW_SIZE*10 + i still fixes the problem. Therefore your model is overfitting because training acc is 1 and testing isn't. Try with changing batch size and learning rate. Also, use optimizer=optimizer instead of 'adam' – nishu b Jun 21 at 7:22