Suppose we're training a recurrent neural net (RNN) on a single, long time series using truncated backpropagation through time (BPTT). We make repeated sweeps through the time series, updating parameters with BPTT multiple times per sweep, as described by Sutskever (2013):
Truncated BPTT...processes the sequence one timestep at a time, and every $k_1$ timesteps, it runs BPTT for $k_2$ timesteps, so a parameter update can be cheap if $k_2$ is small. Consequently, its hidden states have been exposed to many timesteps and so may contain useful information about the far past, which would be opportunistically exploited.
He gives the pseudo-code:
for t from 1 to T do Run the RNN for one step if t divides k_1 then Run BPTT, from t down to t - k_2 end if end for
My question is about how to choose the time points where BPTT is run. If we always start the sweep at time 0 and $k_1$ is fixed (and >1), we'd always end up running BPTT at the same subset of time points. Is this ok, or could it introduce some kind of undesirable regularity (like overweighting certain points relative to others)? Is there any need to start BPTT from different time points on each sweep (e.g. by starting each sweep at a random offset, or randomizing $k_1$)?