Skip to main content
Cross Validated

Return to Answer

added 112 characters in body
Source Link
Franck Dernoncourt
Franck Dernoncourt
  • 47.6k
  • 33
  • 179
  • 293

I had wondered the same thing, and it looks like this has been thoroughly researched in this nice blog post:

http://r2rt.com/styles-of-truncated-backpropagation.html (mirror)

The author found that the disjoint approach, which is what TensorFlow uses, works reasonably well. Using a sliding window every time step is difficult/expensive to calculate (especially with a framework like TensorFlow), and doesn't yield much benefit. The post cites a paper from the 1990s that found jumping forward h steps, then running BPTT back 2h steps was similar in outcome to just doing the disjoint approach of jumping h steps and running BPTT back also h steps.

If you're using LSTM's instead of simple RNN cells, then longer contexts do matter and it would probably be better not to truncate at all, or at least use a really high number of unroll steps. The author's results were for a simple RNN model.

I had wondered the same thing, and it looks like this has been thoroughly researched in this nice blog post:

http://r2rt.com/styles-of-truncated-backpropagation.html

The author found that the disjoint approach, which is what TensorFlow uses, works reasonably well. Using a sliding window every time step is difficult/expensive to calculate (especially with a framework like TensorFlow), and doesn't yield much benefit. The post cites a paper from the 1990s that found jumping forward h steps, then running BPTT back 2h steps was similar in outcome to just doing the disjoint approach of jumping h steps and running BPTT back also h steps.

If you're using LSTM's instead of simple RNN cells, then longer contexts do matter and it would probably be better not to truncate at all, or at least use a really high number of unroll steps. The author's results were for a simple RNN model.

I had wondered the same thing, and it looks like this has been thoroughly researched in this nice blog post:

http://r2rt.com/styles-of-truncated-backpropagation.html (mirror)

The author found that the disjoint approach, which is what TensorFlow uses, works reasonably well. Using a sliding window every time step is difficult/expensive to calculate (especially with a framework like TensorFlow), and doesn't yield much benefit. The post cites a paper from the 1990s that found jumping forward h steps, then running BPTT back 2h steps was similar in outcome to just doing the disjoint approach of jumping h steps and running BPTT back also h steps.

If you're using LSTM's instead of simple RNN cells, then longer contexts do matter and it would probably be better not to truncate at all, or at least use a really high number of unroll steps. The author's results were for a simple RNN model.

Source Link
James D
James D
  • 141
  • 3

I had wondered the same thing, and it looks like this has been thoroughly researched in this nice blog post:

http://r2rt.com/styles-of-truncated-backpropagation.html

The author found that the disjoint approach, which is what TensorFlow uses, works reasonably well. Using a sliding window every time step is difficult/expensive to calculate (especially with a framework like TensorFlow), and doesn't yield much benefit. The post cites a paper from the 1990s that found jumping forward h steps, then running BPTT back 2h steps was similar in outcome to just doing the disjoint approach of jumping h steps and running BPTT back also h steps.

If you're using LSTM's instead of simple RNN cells, then longer contexts do matter and it would probably be better not to truncate at all, or at least use a really high number of unroll steps. The author's results were for a simple RNN model.