Stackable LSTM layer trained with arbitrary BPTT time steps

Anyone knows how to make a LSTM layer that is able to be trained with arbitrary BPTT time steps and easy to be stacked together?

I am now implementing a basic version of LSTM layer.

My scan function looks like:

sequences=self.inputs,
outputs_info=[c0, h0],
non_sequences=[self.Wxi, self.Whi, self.Wci, self.bi,
self.Wxf, self.Whf, self.Wcf, self.bf,
self.Wxc, self.Whc, self.bc,
self.Wxo, self.Who, self.Wco, self.bo
],
n_steps=n_steps
)
self.outputs = hh

where __step_fprop is a function that describes the internal mechanism of LSTM defined as following.

def __step_fprop(u_t, c_tm1, h_tm1,
Wxi, Whi, Wci, bi,
Wxf, Whf, Wcf, bf,
Wxc, Whc, bc,
Wxo, Who, Wco, bo,
):
# input gate
ig = T.nnet.sigmoid(T.dot(u_t, Wxi) +
T.dot(h_tm1, Whi) +
T.dot(c_tm1, Wci) +
bi)
# forget gate
fg = T.nnet.sigmoid(T.dot(u_t, Wxf) +
T.dot(h_tm1, Whf) +
T.dot(c_tm1, Wcf) +
bf)

# cell
cc= fg * c_tm1 + ig * T.tanh(T.dot(u_t, Wxc) +
T.dot(h_tm1, Whc) +
bc)
#  output gate
og = T.nnet.sigmoid(T.dot(u_t, Wxo) +
T.dot(h_tm1,Who)  +
T.dot(c_tm1, Wco) +
bo)
# hidden state
hh = og * T.tanh(cc)

return cc, hh

If I fix input dimension to be 1 and the n_steps to be 3. I must input narray of length 3 to the train function.

However if I specify this layer's BPTT with 3 time steps then this layer will always output the 3 units and it is not convenient to connect to next layer.

A solution to this is to make a whole model and using one scan and specify how many steps it would like to go back through time for whole model but in this case it is not convenient to stack layers together.

Anyone knows how to make a LSTM layer that is able to be trained with arbitrary BPTT time steps and easy to be stacked together?

Kyung Hyun Cho used one scan in each layer for his NMT so I guess it is possible to use scan for each layer.

I think what you're asking for is to train masking with an LSTM and then using the LSTM in a stateful manner for deployment. Since you asked this question many years ago, this has become quite simple.

In this question, I show how to do this in two ways with Keras. I have copied the easiest way below for your reference:

from keras.models import Sequential
from keras.layers import LSTM, Dense
import numpy as np

def gen_sig(num_samples, seq_len):
one_indices = np.random.choice(a=num_samples, size=num_samples // 2, replace=False)

x_val = np.zeros((num_samples, seq_len), dtype=np.bool)
x_val[one_indices, 0] = 1

y_val = np.zeros(num_samples, dtype=np.bool)
y_val[one_indices] = 1

return x_val, y_val

N_train = 100
N_test = 10
recall_len = 20

X_train, y_train = gen_sig(N_train, recall_len)

X_test, y_test = gen_sig(N_train, recall_len)

print('Build STATEFUL model...')
model = Sequential()
model.add(LSTM(10, batch_input_shape=(1, 1, 1), return_sequences=False, stateful=True))

print('Train...')
for epoch in range(15):
mean_tr_acc = []
mean_tr_loss = []

for seq_idx in range(X_train.shape):
start_val = X_train[seq_idx, 0]
assert y_train[seq_idx] == start_val
assert tuple(np.nonzero(X_train[seq_idx, :])).shape == start_val

y_in = np.array([y_train[seq_idx]], dtype=np.bool)

for j in range(np.random.choice(a=np.arange(5, recall_len+1))):
x_in = np.array([[[X_train[seq_idx][j]]]])
tr_loss, tr_acc = model.train_on_batch(x_in, y_in)

mean_tr_acc.append(tr_acc)
mean_tr_loss.append(tr_loss)

model.reset_states()

print('accuracy training = {}'.format(np.mean(mean_tr_acc)))
print('loss training = {}'.format(np.mean(mean_tr_loss)))
print('___________________________________')

mean_te_acc = []
mean_te_loss = []
for seq_idx in range(X_test.shape):
start_val = X_test[seq_idx, 0]
assert y_test[seq_idx] == start_val
assert tuple(np.nonzero(X_test[seq_idx, :])).shape == start_val

y_in = np.array([y_test[seq_idx]], dtype=np.bool)

for j in range(np.random.choice(a=np.arange(5, recall_len+1))):
te_loss, te_acc = model.test_on_batch(np.array([[[X_test[seq_idx][j]]]], dtype=np.bool), y_in)
mean_te_acc.append(te_acc)
mean_te_loss.append(te_loss)
model.reset_states()

print('accuracy testing = {}'.format(np.mean(mean_te_acc)))
print('loss testing = {}'.format(np.mean(mean_te_loss)))
print('___________________________________')