5
$\begingroup$

What are the criteria used to choose between plain vanilla RNN and LSTM RNN when you have to model a generic time series?

$\endgroup$
  • $\begingroup$ LSTM is an architecture that solves the vanishing gradient problem of plain vanilla RNN, so unless there are other considerations, there is no reason not to choose LSTM. $\endgroup$ – horaceT Jul 28 '16 at 20:36
  • $\begingroup$ @horaceT from my understanding LSTM helps the model keeping track of older datapoints better than the plain vanilla model. Is this correct? What about when there is a not so strong dependence on the oldest datapoints compared to the newest? $\endgroup$ – mickkk Jul 28 '16 at 20:42
  • $\begingroup$ The big deal about RNN is its memory capability for modeling sequential patterns. But before LSTM was invented, it was plagued with gradients that die after a few steps. $\endgroup$ – horaceT Jul 28 '16 at 20:47
4
$\begingroup$

Empirically. The criteria is the performance on the validation set. Typically LSTM outperforms RNN, as it does a better job at avoiding the vanishing gradient problem, and can model longer dependences. Some other RNN variants sometimes outperform LSTM for some tasks, e.g. GRU.


FYI:

  • Greff, Klaus, Rupesh Kumar Srivastava, Jan Koutník, Bas R. Steunebrink, and Jürgen Schmidhuber. "LSTM: A search space odyssey." arXiv preprint arXiv:1503.04069 (2015).: "In this paper, we present the first large-scale analysis of eight LSTM variants on three representative tasks: speech recognition, handwriting recognition, and polyphonic music modeling. The hyperparameters of all LSTM variants for each task were optimized separately using random search and their importance was assessed using the powerful fANOVA framework".
  • Zaremba, Wojciech. Ilya Sutskever. Rafal Jozefowicz "An empirical exploration of recurrent network architectures." (2015): used evolutionary computation to find optimal RNN structures.
  • Bayer, Justin, Daan Wierstra, Julian Togelius, and Jürgen Schmidhuber. "Evolving memory cell structures for sequence learning." In International Conference on Artificial Neural Networks, pp. 755-764. Springer Berlin Heidelberg, 2009.: used evolutionary computation to find optimal RNN structures.
  • Le, Quoc V., Navdeep Jaitly, and Geoffrey E. Hinton. "A simple way to initialize recurrent networks of rectified linear units." arXiv preprint arXiv:1504.00941 (2015): shows that RNNs can sometime have performances similar to LSTMs when the identity matrix is used to initialize the recurrent weight matrix.
$\endgroup$
  • $\begingroup$ Empirically and/or "brute force testing" methods seem very common as I dive deeper in machine learning :). But that makes sense. Thanks. $\endgroup$ – mickkk Jul 28 '16 at 20:44
  • 1
    $\begingroup$ @Frank Can you cite a paper where plain RNN outperforms LSTM? $\endgroup$ – horaceT Jul 28 '16 at 20:44
  • $\begingroup$ @horaceT nothing off the top of my head, but since LSTM has more parameters I'd guess there could be some corner cases where RNNs perform better. In practice, personally I directly try LSTM/GRU. $\endgroup$ – Franck Dernoncourt Jul 28 '16 at 20:45
  • $\begingroup$ @Franck I read Zaremba a while back, it shows there are other variants of LSTM that performs better. Thanks for pointing to this paper. $\endgroup$ – horaceT Jul 28 '16 at 20:50
  • $\begingroup$ @horaceT Le, Quoc V., Navdeep Jaitly, and Geoffrey E. Hinton. "A simple way to initialize recurrent networks of rectified linear units." arXiv preprint arXiv:1504.00941 (2015). shows that RNNs can sometime have performances similar to LSTMs when the identity matrix is used to initialize the recurrent weight matrix. $\endgroup$ – Franck Dernoncourt Jul 31 '16 at 23:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.