4
$\begingroup$

Suppose we're training a recurrent neural net (RNN) on a single, long time series using truncated backpropagation through time (BPTT). We make repeated sweeps through the time series, updating parameters with BPTT multiple times per sweep, as described by Sutskever (2013):

Truncated BPTT...processes the sequence one timestep at a time, and every $k_1$ timesteps, it runs BPTT for $k_2$ timesteps, so a parameter update can be cheap if $k_2$ is small. Consequently, its hidden states have been exposed to many timesteps and so may contain useful information about the far past, which would be opportunistically exploited.

He gives the pseudo-code:

for t from 1 to T do
    Run the RNN for one step
    if t divides k_1 then
        Run BPTT, from t down to t - k_2
    end if
end for

My question is about how to choose the time points where BPTT is run. If we always start the sweep at time 0 and $k_1$ is fixed (and >1), we'd always end up running BPTT at the same subset of time points. Is this ok, or could it introduce some kind of undesirable regularity (like overweighting certain points relative to others)? Is there any need to start BPTT from different time points on each sweep (e.g. by starting each sweep at a random offset, or randomizing $k_1$)?

$\endgroup$

1 Answer 1

2
$\begingroup$

Whether or not truncated BPTT is appropriate depends on your application. For language modeling, you could back-propagate every eight words and probably not have to worry about the undesirable regularity that you mentioned. The truncated BPTT will save time during training from having to back-propagate all the way to the beginning of a really long sentence. Most of the useful information for language modeling occurs in the words that are in the recent history.

If you're working with a different type of data you will have to make a judgment about if the truncation is appropriate or not. For applications that require memorizing information over a long time span then truncated BPTT is definitely not appropriate.

$\endgroup$
1
  • $\begingroup$ Thanks for your answer. I understand the issues about whether or not to use truncation. I'm more wondering: given that we're repeatedly sweeping though a single, long sequence, does it matter if the truncated BPTT steps always happen at the same points in that sequence? $\endgroup$
    – user20160
    May 27, 2016 at 3:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.