I've been using theano to experiment with LSTMs, and was wondering what optimization methods (SGD, Adagrad, Adadelta, RMSprop, Adam, etc) work best for LSTMs? Are there any research papers on this topic?
Also, does the answer depend on the type of application I am using the LSTM for? If so, I am using LSTMs for text classification (where the text is first converted into word vectors).
Finally, would the answers be the same or different for RNNs? Any pointers to research papers, or personal insight would be highly appreciated!
LSTMs seem to be quite powerful and I am interested in learning more about how to best use them.