Truncated Back-propagation through time for RNNs

Question

I am not very clear on what is the proper way to train an RNN. Suppose we are using a vanilla RNN and are given some categorical sequence $x$ of length $T$:

$$x= [ x_1,\ldots,x_T]$$

To fit the parameters, compute a cross-entropy loss as follows. First, I compute the network outputs at all time-steps:

$$h_t = tanh(W_{hx}\, x_t + W_{hh} \, h_{t-1})\\ y_t = W_{yh} \, h_t \quad \text{for} \quad t < T $$

Each output is passed through a softmax to get a distribution vector $P_t$ over the categorical output space. I use the loss

$$-\frac{1}{T-1}\sum_{t=1}^{T-1} \log P_t[x_{t+1}]$$

For some toy models, i.e. small T, training this model by unfolding the RNN over the whole time series and doing SGD works well. I believe that this is what is referred to as back-propagation through time (BPTT).

Now, for large $T$ I would like to use truncation, i.e. essentially only consider the past up to some point $k \ll T$, say $k=30$. I tried two variations, but do not see a clear winner:

• the first approach is to split my sequence $x$ into disjoint sub-sequences of length $k$ and then doing a descent step for each of these sub-sequences
• second, I tried a sliding window approach, where the window size is $k$, thus processing all $O(T)$ sub-sequences of length $k$

Also, one could imagine intermediate methods, where the sliding window overlaps by some larger amount.

Does anyone have experience with this or knows of some tutorials or reviews about this?

Thanks!

Answer 1

I had wondered the same thing, and it looks like this has been thoroughly researched in this nice blog post:

http://r2rt.com/styles-of-truncated-backpropagation.html (mirror)

The author found that the disjoint approach, which is what TensorFlow uses, works reasonably well. Using a sliding window every time step is difficult/expensive to calculate (especially with a framework like TensorFlow), and doesn't yield much benefit. The post cites a paper from the 1990s that found jumping forward h steps, then running BPTT back 2h steps was similar in outcome to just doing the disjoint approach of jumping h steps and running BPTT back also h steps.

If you're using LSTM's instead of simple RNN cells, then longer contexts do matter and it would probably be better not to truncate at all, or at least use a really high number of unroll steps. The author's results were for a simple RNN model.