I would like to understand better why LSTM can remember information for a longer time period than vanilla/simple recurrent neural network (SRNN) by redoing an experiment from the paper Learning Long-Term Dependencies with Gradient Descent is Difficult by Bengio et al. 1994.
See Fig 1. and 2 on that paper. The task is simple, given a sequence, if it starts with a high value (e.g. 1), then the output label is 1; if it starts with a low value (e.g. -1), then the output label is 0. The middle is noise. This task is called information latching since the model needs to remember the starting value while going through the middle noise in order to output a correct label. It used a single neuron RNN to build a model that exhibit such behavior. The Figure 2(b) shows the results, and the success frequency of training such a model decreases dramatically as the sequence length increases. There was no result for LSTM since it was wasn't invented yet in 1994.
So, I become curious and would like to see that if LSTM would indeed perform better for such a task. Similarly, I constructed a single neuron RNN for both vanilla and LSTM cell to model information latching. Surprisingly, I found LSTM performs worse, and I don't know why. Could anyone help me explain or if there is anything wrong with my code, please?
Here is my result:
Here is my code:
import matplotlib.pyplot as plt
import numpy as np
from keras.models import Model
from keras.layers import Input, LSTM, Dense, SimpleRNN
N = 10000
num_repeats = 30
num_epochs = 5
# sequence length options
lens = [2, 5, 8, 10, 15, 20, 25, 30] + np.arange(30, 210, 10).tolist()
res = {}
for (RNN_CELL, key) in zip([SimpleRNN, LSTM], ['srnn', 'lstm']):
res[key] = {}
print(key, end=': ')
for seq_len in lens:
print(seq_len, end=',')
xs = np.zeros((N, seq_len))
ys = np.zeros(N)
# construct input data
positive_indexes = np.arange(N // 2)
negative_indexes = np.arange(N // 2, N)
xs[positive_indexes, 0] = 1
ys[positive_indexes] = 1
xs[negative_indexes, 0] = -1
ys[negative_indexes] = 0
noise = np.random.normal(loc=0, scale=0.1, size=(N, seq_len))
train_xs = (xs + noise).reshape(N, seq_len, 1)
train_ys = ys
# repeat each experiments multiple times
hists = []
for i in range(num_repeats):
inputs = Input(shape=(None, 1), name='input')
rnn = RNN_CELL(1, input_shape=(None, 1), name='rnn')(inputs)
out = Dense(2, activation='softmax', name='output')(rnn)
model = Model(inputs, out)
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
hist = model.fit(train_xs, train_ys, epochs=num_epochs, shuffle=True, validation_split=0.2, batch_size=16, verbose=0)
hists.append(hist.history['val_acc'][-1])
res[key][seq_len] = hists
print()
fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(pd.DataFrame.from_dict(res['lstm']).mean(), label='lstm')
ax.plot(pd.DataFrame.from_dict(res['srnn']).mean(), label='srnn')
ax.legend()
I also have the result shown in notebook, which would be convenient if you'd like to replicate the results. It took over a day to run the experiment on my machine using CPU only. It could be faster on a GPU-enabled machine.
Update 2018-04-18:
I tried to reproduce a figure on the landscape of RNN inspired by Figure 6 in On the difficulty of training Recurrent Neural Networks. I find it interesting seeing the formation of cliff in the loss landscape as the number of recurrence/time steps/sequence length increases, which could be related related to explaining the difficult of training long sequences observed here. More details is available here.
Update 2018-04-19
Extending @shimao's experiment. It seems that LSTM and GRU are just not so good at latching on information. But switching to a different task, which I call bit relay, (@shimao's task 2), GRU performs better while SRNN and LSTM are equally bad.
Now, I tend to think the performance of a cell-type could be task-specific.
Task 1: information latching (1 unit; 10 repeats; 10 epochs)
Task 2: bit relay (8 unit; 10 repeats; 10 epochs)
Error bars are standard deviations.
Then, an intriguing question is why LSTM doesn't work on information latching. Given the simplicity of the task, it should be able to work, shouldn't it? Could be related to the landscape (e.g. Cliffs) with respect to its gradients.