1
$\begingroup$

A common way to train Recurrent Neural Networks (RNNs) is back-propagation through time (BPTT), whereby the entire computational graph is 'unrolled' along each timestep and the derivatives can be calculated normally.

For RNNs operating on long sequences, this creates a very large graph, and training can be unfeasibly slow. To counteract this, assuming one can't perform any tricks like splitting the sequences, it is common to use Truncated BPTT, whereby the derivative is only calculated every $k_1$ steps, and the computational graph is only unrolled for $k_2$ steps at each update.

As far as I understand it, a requirement of TPTT, is that one can keep the state of the network from earlier than $k_2$, such that one doesn't have to re-calculate earlier results repeatedly. Therefore, it is not clear how it is possible to perform such a procedure efficiently for a bidirectional recurrent network. I could maybe conceive of using a clever state caching system to do this, whereby the reverse networks state is calculated in chunks up front and matched up with the forward when the time is right, but this has problems:

  1. It seems like it could take a lot of memory, as one will have to cache states for the entire length of the sequence
  2. The state for each sequence will be calculated with weights that won't change until the example is complete.

Is this a problem that people have encountered? If you don't know explicitly, then what is the longest Bidirectional Recurrent topology that you know of?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.