RNNs and their variations use the previous time-step output and input, but has there ever been any research done into using multiple time-steps (ie, using each time-step going 10 time steps back) ?
Is there a reason why such models are not used?
$\begingroup$
$\endgroup$
asked Dec 13, 2017 at 0:20
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
RNNs are directly connected to only the previous time step. However, because the hidden state at that time-step ($t-1$) was influenced by the hidden state at $t-2$, and so on, the RNN can already effectively look back any number of time steps in an indirect manner.
You might try to make connections from time $t-k, t-k+1, t-k+2, \ldots, t-1$ to time $t$ but that has the issue of adding a large number of parameters to the model, which is both computationally expensive and induces overfitting.
But there are similar approaches to going "multiple time-steps" back in time. Attention mechanisms and Wavenet come to mind. Wavenet isn't a recurrent network, but it uses dilated convolutions to use many previous time-steps. Attention mechanism allows the RNN to look at all previous time steps simultaneously.
https://distill.pub/2016/augmented-rnns/ https://deepmind.com/blog/wavenet-generative-model-raw-audio/
