I'm new in area of deep learning and for me first step was to read interesting articles from deeplearning.net site. In papers about deep learning, Hinton and others mostly talk about applying it to image problems. Can someone try to answer me can it be applied to problem of predicting time series values (financial, internet traffic,...) and what are important things that I should focus if it is possible?
$\begingroup$ Do you have any matlab code example for using deep learning for forecasting? $\endgroup$– user3209559Feb 14, 2014 at 8:49
$\begingroup$ No, I'm using code example from deeplearning.net and pylearn2 library which I modified. Try to find matlab code examples on this page and try to make modifications necessary for forecasting. $\endgroup$– VedranFeb 18, 2014 at 9:02
There has been some work on adapting deep learning methods for sequential data. A lot of this work has focused on developing "modules" which can be stacked in a way analogous to stacking restricted boltzmann machines (RBMs) or autoencoders to form a deep neural network. I'll highlight a few below:
- Conditional RBMs: Probably one of the most successful applications of deep learning for time series. Taylor develops a RBM like model that adds temporal interactions between visible units and apply it to modeling motion capture data. Essentially you end up with something like a linear dynamical system with some non-linearity added by the hidden units.
- Temporal RBMs: In his thesis (section 3) Ilya Sutskever develops several RBM like models with temporal interactions between units. He also presents some interesting results showing training recurrent neural networks with SGD can perform as well or better than more complex methods, like Martens' Hessian-free algorithm, using good initialization and a slightly modified equation for momentum.
- Recursive Autoencoders: Lastly I'll mention the work of Richard Socher on using recursive autoencoders for parsing. Although this isn't time series, it is definitely related.
1$\begingroup$ There's also a notion floating around in the deep/feature learning community that you can look at a recurrent network (often applied to sequential data) as a deep network "on its side," where all layers share the same weight matrix. $\endgroup$– lmjohns3Oct 6, 2013 at 21:26
$\begingroup$ What about the requirement on the computational power for using deep learning? Is Python or Matlab suitable for solving industry-strength problem of deep learning? $\endgroup$– user3269Mar 5, 2014 at 13:25
$\begingroup$ Pylearn2 use theano library which enables execution on GPUs with CUDA support. $\endgroup$– VedranMar 10, 2014 at 8:59
Yes, deep learning can be applied for time series predictions. In fact, it has been done many times already, for example:
This is not really any "special case", deep learning is mostly about preprocessing method (based on generative model), so to you have to focus on exactly same things that you focus on when you do deep learning in "traditional sense" on one hand, and same things you focus on while performing time series predictions without deep learning.
Recurrent Neural Networks are considered a type of Deep Learning (DL). I think they are the most popular DL tool for (1d) sequence-to-sequence learning. They are currently the basis of Neural Machine Translation (NMT) approaches (pioneered 2014 at LISA (UdeM), Google, and probably a couple others I'm not remembering).
Alex Graves' Generating sequences with Recurrent Neural Networks uses recurrent networks and Long short term memory Cells to predict text and do handwriting synthesis.
Andrej Karpathy has written a blog about generating character level sequences from scratch. He uses RNNs in his tutorial.
For more examples, you should look at- Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780.
Maybe this will help:
- I. Sutskever, O. Vinyals, and Q. V. V Le, “Sequence to sequence learning with neural networks,” in Advances in Neural Information Processing Systems, 2014, pp. 3104–3112.
If you have definition for your exact time window on the data like sentences in this paper or paragraphs then you will be fine with using LSTM, but I am not sure how to find the time window that are not obvious and are more context aware. An example for that can be how many of log data you are seeing are related and that's not something obvious.
1$\begingroup$ Could you maybe say what in the paper is helpful? This is interesting stuff $\endgroup$ Dec 22, 2014 at 15:23
1$\begingroup$ I think you can find some other papers in here: cs.stanford.edu/people/ang/?page_id=414 $\endgroup$– M.RezDec 23, 2014 at 11:13
5$\begingroup$ thanks but that's not what I meant. Usually here we ask that people explain how papers they link to are relevant to the answer. It's helpful to everybody, who might not have time to track down and read the paper, and especially to people without library database subscriptions who can't get past paywalls $\endgroup$ Dec 23, 2014 at 15:28