Cannot understand LSTM inference

Question

I seem to have stumbled on a hole in my understanding around LSTMs. In short, I cannot understand how even a simple one is actually fed samples, upon inference time/training time. Here are the details:

• I have time-series weather data of temperature, measured every hour. (Not important, but just so that you have context)

• During inference time, at every point in the present, I want to predict the next 5 time-steps in the future.

I know that there is a strong long-term dependency of about 24 samples in the past. (Meaning, something 24 samples in the past, is very informative to things going on now).

So my questions are:

Inference time: I have weather data coming in continuously, one sample at a time. From the present time, I want to predict 5 samples into the future. I would like to know: Literally, at every time step, what am I inputting to the LSTM? Am I inputting just the next sample, and then letting it predict 5 future samples? Am I inputting the current new sample AND say, 10 past samples, and letting it output the next 5 future samples? In either case, how does it keep track of a very long-term dependency, from 24 samples ago?

Training time: Once again, I have a time-series of weather data, say it is 1,000,000 samples long. How would I train an LSTM here, to do what I described above? I am asking, what - mechanically - am I inputting into the LSTM, at every training iteration... literally, during each training iteration, what should the input be, such that I capture strong dependencies from 24 samples ago?

Thank you!

Answer 1

The LSTM prediction $y^{(t)}$ is a function $f$ of the previous hidden state data $h^{(t-1)}$ and the previous step's observed data $x^{(t-1)}$.

$$ \hat{x}^{(t)}, h^{(t)} = f\left(x^{(t-1)}, h^{(t-1)}\right) $$

To model dependencies $k$ steps in the past, you need to compute your hidden state to be inclusive of that long-term data. First, initialize the hidden state (usually people initialize with 0s). Then, compute the hidden state $k$ steps into the future using the observed $x$ values and your newly-computed hidden values. At step $k$, you have a hidden state which has been updated to store data about steps $0, 1, ..., k-1$.

The key idea is that $h^{(t)}$ contains all relevant data about the past, and $x^{(t-1)}$ tells you about the present. If the model is any good, then $f\left(x^{(t-1)}, h^{(t-1)}\right)$ will be a reasonable prediction about the future.

You can make 1-step-ahead predictions using the observed $x$. If you need $m$-step-ahead predictions, then you'll have to use your predicted $\hat{x}$s as the inputs instead of observed values $x$.

You always need to have 2 things to use an LSTM: the new observed data, and the hidden state. This is true during training time and testing time. To train an LSTM, the simplest method is to start at the beginning of your training data (the earliest data) and move forward, back-propagating at each time step. Unlike for an MLP, you can't shuffle the data because you need the hidden state from $t-1$, and that hidden state is dependent on the previous time step and so on back to the start of your time series.

The term of art for RNN training is back-propagation through time (BPTT).

There are a few different "twists" on this basic idea. Two are mini-batches and using longer time sequences than 1 step.

Using mini-batches can improve computation time. The way to do this is to divide your time-series into some number of contiguous, equal-length chunks. You'll lose a little information when predicting at the beginning of each chunk. This is because the time steps on the boundaries of chunks won't be able to "see" the true hidden state because at the start of training you'll be initializing them. (See also: What happens to the initial hidden state in an RNN layer?) The basic idea is that instead of 1 time series containing 4 years of data, you can divide it up to have 4 time series containing 1 year each.

Using longer sequences helps because the parameters of the model see data from more than 1 time step, and can assign credit/blame for predictions. But it's hard to retain data from hundreds or more of time-steps in the past (vanishing gradient). Using longer sequences means taking $n$ time steps at a time and also predicting $n$ time steps at a time, and back-propagating for all $n$ in one step. So the input observed data is time steps $$t, t+1, t+2, ..., t+n$$ and predicting time steps $$t+1,t+2, ..., t+n+1.$$ Then you advance your offset by $n$ so that the hidden state matches the correct time step.