2
$\begingroup$

I'm trying to learn more about RNNs and I'm tackling a toy problem. I'm generating some data that has a pattern, two 1s followed by three 0s which keeps repeating infinitely without any noise. So my master data is something like [1 1 0 0 0 1 1 0 0 0 1 1 0 0 0 1 ... ]

Then I slide a window of N timesteps over the data and feed this into an LSTM, asking it to predict the next value. I'm treating this as a binary classification problem.

model = tf.keras.models.Sequential([
    tf.keras.layers.LSTM(4, input_shape=(None, 1)),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

model.fit(train_gen, epochs=EPOCHS)

All is well, the model reaches 100% accuracy and near 0 loss pretty quickly. However, unexpected things start happening when I feed in sequences of different lengths (drawn from the same master data).

for i in range(15):
    TEST_WINDOW_SIZE = WINDOW_SIZE + i

    longer_data_gen = TimeseriesGenerator(train_data[:2000], train_data[:2000], TEST_WINDOW_SIZE, batch_size=2000)

    [loss, acc] = model.evaluate(longer_data_gen)
    if acc < 1.0:
        print('For i={} accuracy is {}'.format(i, acc))

The output will be something like

i = 0, acc = 1.0
i = 1, acc = 0.6
i = 2, acc = 0.2
i = 3, acc = 0.2
i = 4, acc = 0.6
i = 5, acc = 1.0
...

So basically the network learned the pattern, but it isn't syncing it to the input, it's off-phase.

Note:

  • In my experiments, adding dropout=0.15 to the LSTM sometimes fixes the problem, depending on the run, however the reported accuracy on the training set doesn't get to 100%, despite me getting 100% accuracy on all my variable length test data.

  • I also tried lowering the number of hidden units in the LSTM but it doesn't seem to do the job

  • Sometimes it generalizes even without dropout, but most of the time it doesn't

I kind of get the feeling I'm doing something wrong here, it seems like it's pretty hard to generalize on such a simple problem. Am I approaching this wrong ?

The full code is here.

$\endgroup$
1
$\begingroup$

It seems LSTM hasn't really learnt the short term pattern, which is the only pattern. If you see carefully, it only needs last 3 inputs to "guess", or in your case "predict" (because you used sigmoid) the next number. For instance

i/p    o/p
11000 - 1
01100 - 0
00110 - 0
00011 - 0
10001 - 1

But your LSTM is relying on values that are even further from last 3 digits to predict the next digit. A simple fix of

TEST_WINDOW_SIZE = WINDOW_SIZE*10 + i

should give you an accuracy of 1, irrespective of value of i

Also, the model is overfit. If you get an accuracy of 0.9<acc<1.0 by 5th iteration, then probably the fix won't be needed

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I don't quite understand. training_example_length == WINDOW_SIZE. Unless you mean pattern length ? My pattern length is 5 and WINDOW_SIZE=10. How is WINDOW_SIZE much larger ? In practice, if I know the length of my pattern then what use is an RNN to me? The network should learn the pattern and I feel like I give it enough context for it to be able to. $\endgroup$ – Adrian Buzea Jun 19 at 16:03
  • $\begingroup$ yes, oversight from me, I thought training_example_length = 32. That said, TEST_WINDOW_SIZE = WINDOW_SIZE*10 + i still fixes the problem. Therefore your model is overfitting because training acc is 1 and testing isn't. Try with changing batch size and learning rate. Also, use optimizer=optimizer instead of 'adam' $\endgroup$ – nishu b Jun 21 at 7:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.