Stateful LSTM for time-series prediction - should each input sequence be shifted by 1 time step or sequenceLength time steps

Question

I am building an LSTM, to attempt to learn the trend historic trend of some time-series data set (e.g. the daily share price of a company). When training my network, I am taking batches of size 1, each consisting of 25 sequential daily closing prices, where it then makes a prediction for the price on the 26th day.

Say the first sequence in an epoch used to train begins at t=0 and ends at t=24. I am using a stateful LSTM and hence taking the output state of one batch and inputting it to the state of the next batch, does this mean that my next batch must be [t=25, t=49]? May I instead "slide" each batch by 1 time step so that the 2nd batch is [t=1, t=25], or does this defeat the purpose of passing the state between batches?

Answer 1

You can't slide each batch by 1 time step, if the batch sequence length is longer than 1 time step, and if you are preserving the state between batches.

Your implied question is "how can I train on other sequences that start at time-steps that are not multiples of the sequence length?".

There are a few options:

• reset the state at each batch. This reduces the ability of the network to use information from earlier than the current sequence. But it's easy to do, and if your sequence length is long enough (20-50?), then anything much longer than that will likely get 'gradient vanished' away anyway
• start at a different offset at the start for each epoch. So, all sequences in the epoch will be offset slightly
• use randomized sequence lengths for each batch, so they finish a bit earlier/later