What optimization methods work best for LSTMs?

Question

I've been using theano to experiment with LSTMs, and was wondering what optimization methods (SGD, Adagrad, Adadelta, RMSprop, Adam, etc) work best for LSTMs? Are there any research papers on this topic?

Also, does the answer depend on the type of application I am using the LSTM for? If so, I am using LSTMs for text classification (where the text is first converted into word vectors).

Finally, would the answers be the same or different for RNNs? Any pointers to research papers, or personal insight would be highly appreciated!

LSTMs seem to be quite powerful and I am interested in learning more about how to best use them.

Answer 1

Ironically the best Optimizers for LSTMs are themselves LSTMs: https://arxiv.org/abs/1606.04474 Learning to learn by gradient descent by gradient descent.

The basic idea is to use a neural network (specifically here a LSTM network) to co-learn and teach the gradients of the original network. It's called meta learning.

This method, while proposed by Juergen Schmidhuber in 2000, was only recently shown to out-perform the other optimizers in RNN training. ( see the original paper for a nice graphic)

Answer 2

There is in general no clear evidence as to which optimisation method to use in what scenario. There has been some analysis in the behaviour of these methods under different scenarios however nothing is conclusive. If you want to dive into this stuff then I recommend: http://papers.nips.cc/paper/5486-identifying-and-attacking-the-saddle-point-problem-in-high-dimensional-non-convex-optimization.pdf

In order to at least provide you with somewhat of an answer I would argue that often the configuration of your optimisation routine is more important than the actual routine itself.

Moreover I recommend you to look into papers to see what techniques are being used. Alex Graves from example has bene using RMSprop throughout most of his publications on generating sequences.