0
$\begingroup$

My question comes from this tutorial about RNNs, but it can be a general RNNs implementation question.

Suppose we want to develop a model to predict the next character using a RNN, and we have the following training data:

X = [A, B, C, D, E, F, G, H]
Y = [B, C, D, E, F, G, H, I]

During training we only consider 1 epoch and process 1 batch at a time, using a sequence length n=4 (number of unrollings). By the referenced tutorial (and even in Karpathy's famous RNN post), this would lead to two training sets:

X_0 = [A, B, C, D]
Y_0 = [B, C, D, E],

X_1 = [E, F, G, H]
Y_1 = [F, G, H, I]

My question is: to capture better the "influence" of the previous n characters, shouldn't the training data be split as

X_0 = [A, B, C, D]
X_1 = [B, C, D, E]
X_2 = [C, D, E, F]
X_3 = [D, E, F, G]
X_4 = [E, F, G, H]

(with corresponding Y's)?

$\endgroup$
0
$\begingroup$

Yes, you can cut sequences as you want, provided that enough information is kept. In your case, model needs to learn 1-step dependency, so for example you can include X = [A, B] related with Y = [B, C] in your training set.

Remark: For time series with long dependence, you need to keep long sequences. In this case training is usually done through stateful LSTM (for example X=[0, 0, ... 0] related with Y = 0, and X=[1, 0, ..., 0] leading to Y=1, as described in http://philipperemy.github.io/keras-stateful-lstm/ )

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.