4
$\begingroup$

As far as I understand the follwoing two models are essentially identical:

  1. Having a stateful LSTM with just a single time step and passing 10 time-series data points into it one by one, and using the final output as the prediction for the 11th data point

  2. Having a stateless LSTM unrolled into 10 time steps passing the entire 10 data points as input to the corresponding time steps, then using the output at the final time step as the prediction for the 11th data point.

Why would you ever need to use version 2 instead of version 1? Although I believe they function identically, version 1 would have benefits such as the ability to use variable input sequence lengths (version 2 would need to have all sequence lengths equal to 10, or else padded with zeros) or be simpler to be used in inference where a single future prediction is made per day, for example.

$\endgroup$

1 Answer 1

7
$\begingroup$

An RNN and any of its more sophisticated versions has a hidden state. The hidden state at time $t$ is a function of the hidden state at time $t-1$ and the input at time $t$. This hidden state at time $0$ is typically initialized to $0$. The fundamental reason why RNNs are "unrolled" is because all previous inputs and hidden states are used in order to compute the gradients wrt the final output of the RNN.

One issue with this is that the memory required to hold all these activations and gradients is linear in the length of the sequence. Therefore, very long sequences require prohibitive amounts of memory. In order to get around this issue, long sequences are typically truncated. For example, if you have 99 words in a phrase and want to predict the 100th, you may only use the last 9.

The idea of stateful RNNs is that you can do better than simply initializing the hidden state at time $0$ to $0$. Indeed, if we are feeding in the 90th word at time $0$, then we should initialize the hidden state as computed by feeding in words $0$ through $89$ in an rolled fashion instead. So we're saying that we're only going to train with shorter sequences, but initialize the hidden state to the same value it would've been computed if we trained with longer sequences.

Obviously, we can't backpropagate through the initial value of the hidden state, since deliberately truncating the backpropagation was the whole point. So your stateful RNN with length 1 would not properly learn relationships across times.

$\endgroup$
2
  • $\begingroup$ +1 for clarity. Just one question: you say that all previous inputs and hidden states are used to compute the gradient of the output $y$ wrt the weights. However, the weights of a LSTM cell (to fix ideas) are independent of time (I think the same is true for most RNN architectures currently in use). So, what you’re saying is that the gradient of $y$ wrt these fixed weights depends on inputs and hidden states at all times. Right? $\endgroup$
    – DeltaIV
    Sep 5, 2018 at 6:56
  • 1
    $\begingroup$ Yes, the computation graph contains multiple inputs and hidden states but only one set of weights. $\endgroup$
    – shimao
    Sep 5, 2018 at 13:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.