What are the criteria used to choose between plain vanilla RNN and LSTM RNN when you have to model a generic time series?
$\begingroup$
$\endgroup$
3
-
$\begingroup$ LSTM is an architecture that solves the vanishing gradient problem of plain vanilla RNN, so unless there are other considerations, there is no reason not to choose LSTM. $\endgroup$– horaceTCommented Jul 28, 2016 at 20:36
-
$\begingroup$ @horaceT from my understanding LSTM helps the model keeping track of older datapoints better than the plain vanilla model. Is this correct? What about when there is a not so strong dependence on the oldest datapoints compared to the newest? $\endgroup$– mickkkCommented Jul 28, 2016 at 20:42
-
$\begingroup$ The big deal about RNN is its memory capability for modeling sequential patterns. But before LSTM was invented, it was plagued with gradients that die after a few steps. $\endgroup$– horaceTCommented Jul 28, 2016 at 20:47
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
6
Empirically. The criteria is the performance on the validation set. Typically LSTM outperforms RNN, as it does a better job at avoiding the vanishing gradient problem, and can model longer dependences. Some other RNN variants sometimes outperform LSTM for some tasks, e.g. GRU.
FYI:
- Greff, Klaus, Rupesh Kumar Srivastava, Jan Koutník, Bas R. Steunebrink, and Jürgen Schmidhuber. "LSTM: A search space odyssey." arXiv preprint arXiv:1503.04069 (2015).: "In this paper, we present the first large-scale analysis of eight LSTM variants on three representative tasks: speech recognition, handwriting recognition, and polyphonic music modeling. The hyperparameters of all LSTM variants for each task were optimized separately using random search and their importance was assessed using the powerful fANOVA framework".
- Zaremba, Wojciech. Ilya Sutskever. Rafal Jozefowicz "An empirical exploration of recurrent network architectures." (2015): used evolutionary computation to find optimal RNN structures.
- Bayer, Justin, Daan Wierstra, Julian Togelius, and Jürgen Schmidhuber. "Evolving memory cell structures for sequence learning." In International Conference on Artificial Neural Networks, pp. 755-764. Springer Berlin Heidelberg, 2009.: used evolutionary computation to find optimal RNN structures.
- Le, Quoc V., Navdeep Jaitly, and Geoffrey E. Hinton. "A simple way to initialize recurrent networks of rectified linear units." arXiv preprint arXiv:1504.00941 (2015): shows that RNNs can sometime have performances similar to LSTMs when the identity matrix is used to initialize the recurrent weight matrix.
answered Jul 28, 2016 at 20:38
-
$\begingroup$ Empirically and/or "brute force testing" methods seem very common as I dive deeper in machine learning :). But that makes sense. Thanks. $\endgroup$– mickkkCommented Jul 28, 2016 at 20:44
-
1$\begingroup$ @Frank Can you cite a paper where plain RNN outperforms LSTM? $\endgroup$– horaceTCommented Jul 28, 2016 at 20:44
-
$\begingroup$ @horaceT nothing off the top of my head, but since LSTM has more parameters I'd guess there could be some corner cases where RNNs perform better. In practice, personally I directly try LSTM/GRU. $\endgroup$ Commented Jul 28, 2016 at 20:45
-
$\begingroup$ @Franck I read Zaremba a while back, it shows there are other variants of LSTM that performs better. Thanks for pointing to this paper. $\endgroup$– horaceTCommented Jul 28, 2016 at 20:50
-
$\begingroup$ @horaceT Le, Quoc V., Navdeep Jaitly, and Geoffrey E. Hinton. "A simple way to initialize recurrent networks of rectified linear units." arXiv preprint arXiv:1504.00941 (2015). shows that RNNs can sometime have performances similar to LSTMs when the identity matrix is used to initialize the recurrent weight matrix. $\endgroup$ Commented Jul 31, 2016 at 23:36