Using deep learning for time series prediction

Question

I'm new in area of deep learning and for me first step was to read interesting articles from deeplearning.net site. In papers about deep learning, Hinton and others mostly talk about applying it to image problems. Can someone try to answer me can it be applied to problem of predicting time series values (financial, internet traffic,...) and what are important things that I should focus if it is possible?

Answer 1

There has been some work on adapting deep learning methods for sequential data. A lot of this work has focused on developing "modules" which can be stacked in a way analogous to stacking restricted boltzmann machines (RBMs) or autoencoders to form a deep neural network. I'll highlight a few below:

• Conditional RBMs: Probably one of the most successful applications of deep learning for time series. Taylor develops a RBM like model that adds temporal interactions between visible units and apply it to modeling motion capture data. Essentially you end up with something like a linear dynamical system with some non-linearity added by the hidden units.
• Temporal RBMs: In his thesis (section 3) Ilya Sutskever develops several RBM like models with temporal interactions between units. He also presents some interesting results showing training recurrent neural networks with SGD can perform as well or better than more complex methods, like Martens' Hessian-free algorithm, using good initialization and a slightly modified equation for momentum.
• Recursive Autoencoders: Lastly I'll mention the work of Richard Socher on using recursive autoencoders for parsing. Although this isn't time series, it is definitely related.

Answer 2

Yes, deep learning can be applied for time series predictions. In fact, it has been done many times already, for example:

• http://cs229.stanford.edu/proj2012/BussetiOsbandWong-DeepLearningForTimeSeriesModeling.pdf
• http://link.springer.com/article/10.1007/s00134-013-2964-2#page-1

This is not really any "special case", deep learning is mostly about preprocessing method (based on generative model), so to you have to focus on exactly same things that you focus on when you do deep learning in "traditional sense" on one hand, and same things you focus on while performing time series predictions without deep learning.

Answer 3

Recurrent Neural Networks are considered a type of Deep Learning (DL). I think they are the most popular DL tool for (1d) sequence-to-sequence learning. They are currently the basis of Neural Machine Translation (NMT) approaches (pioneered 2014 at LISA (UdeM), Google, and probably a couple others I'm not remembering).

Answer 4

Alex Graves' Generating sequences with Recurrent Neural Networks uses recurrent networks and Long short term memory Cells to predict text and do handwriting synthesis.

Andrej Karpathy has written a blog about generating character level sequences from scratch. He uses RNNs in his tutorial.

For more examples, you should look at- Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780.

Answer 5

Maybe this will help:

• I. Sutskever, O. Vinyals, and Q. V. V Le, “Sequence to sequence learning with neural networks,” in Advances in Neural Information Processing Systems, 2014, pp. 3104–3112.

If you have definition for your exact time window on the data like sentences in this paper or paragraphs then you will be fine with using LSTM, but I am not sure how to find the time window that are not obvious and are more context aware. An example for that can be how many of log data you are seeing are related and that's not something obvious.