Let me use daily price prediction of Bitcoin as a simple example (I am not actually working with Bitcoin but its temporal nature fits well to explain my question).
Say I had a data set consisting of the last 101 sequential days of Bitcoin's closing prices
[p1, p2, p3, ... , p101], where
pX is the closing price on day
X. The inputs will be the first 100 days, and the labels will be
[p2, ... , p101] - the inputs shifted by 1 (attempting to predict the next day's closing price).
As I understand how a stateful LSTM works, I could divide my 100 training examples into 4 sequences of 25 examples. Each of these 4 will be a single batch - therefore the input to my LSTM of (batchSize, timeSteps, features) would be (1, 25, 1).
Each epoch would consist of 4 batches. I would first feed in batch1 =
[p1, ... , p25] (and the labels for each time step
[p2, ... , p26]), and pass the final state as the initial state to process batch2 =
[p26, ... , p50], and so on. After all 4 batches are processed, and the epoch is complete, I would then reset the state and repeat for as many epochs as necessary.
If the LSTM could accurately predict the following day's price using the previous 25 days as an input sequence, I would then like to use it to make daily, real-time predictions of prices, not once every 25 days.
It is currently day 101, and I would like to make a prediction for day 102,
p102. I would feed in
[p77, ... , p101] as input. Then tomorrow, on day 102, I would like to predict
p103, so I feed in
[p78, ... , p102]. These batches are no longer continuing on sequentially from each other, but instead are shifted one day forward. How would I deal with the state of the LSTM when doing so? On each of these days would I feed in the previous 100 days as 4 batches of 25 so that the state is built up before I then make my prediction for tomorrow?
In reality, I am working on a much more complex problem with a far more extensive data set. I thought I understood how a stateful LSTM works until I just trained it as explained above in sequential batches. However, I then decided to do this process of shifting each input by one day each batch on the exact same training set. When doing this, the model's accuracy was far lower to what it was during training.
I thought that if I trained a stateful LSTM on 100 examples in 4 batches of 25, I could then take any arbitrary sequence of 25 examples from this same 100 and it would predict the following day with the same accuracy as training.
To make things clearer, here is how my data would be batched to train over 2 epochs, and then make 3 daily predictions after training
Epoch 1 inputs: [p1, ... , p25] [p26, ... , p50] [p51, ... , p75] [p76, ... , p100] Reset State Epoch 2 inputs: [p1, ... , p25] [p26, ... , p50] [p51, ... , p75] [p76, ... , p100]
(Reset State?) (Build up state by processing [p2, ... , p76] in 3 batches of 25?) Inputs to predict price on p102: [p77, ... , p101] (Reset State?) (Build up state by processing [p3, ... , p77] in 3 batches of 25?) Inputs to predict price on p103: [p78, ... , p102] (Reset State?) (Build up state by processing [p4, ... , p78] in 3 batches of 25?) Inputs to predict price on p104: [p79, ... , p103]