What are the correct training batches, given a sequence length of RNNs?

Question

My question comes from this tutorial about RNNs, but it can be a general RNNs implementation question.

Suppose we want to develop a model to predict the next character using a RNN, and we have the following training data:

X = [A, B, C, D, E, F, G, H]
Y = [B, C, D, E, F, G, H, I]

During training we only consider 1 epoch and process 1 batch at a time, using a sequence length n=4 (number of unrollings). By the referenced tutorial (and even in Karpathy's famous RNN post), this would lead to two training sets:

X_0 = [A, B, C, D]
Y_0 = [B, C, D, E],

X_1 = [E, F, G, H]
Y_1 = [F, G, H, I]

My question is: to capture better the "influence" of the previous n characters, shouldn't the training data be split as

X_0 = [A, B, C, D]
X_1 = [B, C, D, E]
X_2 = [C, D, E, F]
X_3 = [D, E, F, G]
X_4 = [E, F, G, H]

(with corresponding Y's)?

Answer 1

Yes, you can cut sequences as you want, provided that enough information is kept. In your case, model needs to learn 1-step dependency, so for example you can include X = [A, B] related with Y = [B, C] in your training set.

Remark: For time series with long dependence, you need to keep long sequences. In this case training is usually done through stateful LSTM (for example X=[0, 0, ... 0] related with Y = 0, and X=[1, 0, ..., 0] leading to Y=1, as described in http://philipperemy.github.io/keras-stateful-lstm/ )