# Truncated Back-propagation through time for RNNs

I am not very clear on what is the proper way to train an RNN. Suppose we are using a vanilla RNN and are given some categorical sequence $x$ of length $T$:

$$x= [ x_1,\ldots,x_T]$$

To fit the parameters, compute a cross-entropy loss as follows. First, I compute the network outputs at all time-steps:

$$h_t = tanh(W_{hx}\, x_t + W_{hh} \, h_{t-1})\\ y_t = W_{yh} \, h_t \quad \text{for} \quad t < T$$

Each output is passed through a softmax to get a distribution vector $P_t$ over the categorical output space. I use the loss

$$-\frac{1}{T-1}\sum_{t=1}^{T-1} \log P_t[x_{t+1}]$$

For some toy models, i.e. small T, training this model by unfolding the RNN over the whole time series and doing SGD works well. I believe that this is what is referred to as back-propagation through time (BPTT).

Now, for large $T$ I would like to use truncation, i.e. essentially only consider the past up to some point $k \ll T$, say $k=30$. I tried two variations, but do not see a clear winner:

• the first approach is to split my sequence $x$ into disjoint sub-sequences of length $k$ and then doing a descent step for each of these sub-sequences
• second, I tried a sliding window approach, where the window size is $k$, thus processing all $O(T)$ sub-sequences of length $k$

Also, one could imagine intermediate methods, where the sliding window overlaps by some larger amount.