Multilayer neural networks for multivariate temporal data I'm looking for a way to model and extract features from multivariate temporal data (e.g., multi-channel audio recordings).
I'm specifically interested in deep learning methods such as RBM, sparse autoencoders and so on.
Most methods I encountered consider only one dimension of the data, or maybe a 2D "block" in the data (usually images).
I couldn't find a paper on how to take into consideration the temporal aspect and also the multivariate aspect. For example if I'm recording audio from multiple channels, then I know that the channels are probably correlated to some degree. Also, since it's an evolving temporal signal, there is also some correlation of each sample to the previous samples.
Is there some way to incorporate these multivariate temporal constraints into a network?
I'm looking for specific papers on methods for  multivariate temporal where the multi-channel structure is used.
Thanks.
 A: There are multiple papers by Hinton et al. which deal with temporal data and also audio (http://www.cs.toronto.edu/~hinton/papers.html). For example:


*

*Acoustic Modeling using Deep Belief Networks, 2012.

*Learning a better Representation of Speech Sound Waves using
Restricted Boltzmann Machines, 2011.

*Deep Belief Networks using Discriminative Features for Phone
Recognition, 2011. The Recurrent Temporal Restricted Boltzmann
Machine, 2009.

*Factored Conditional Restricted Boltzmann Machines for Modeling
Motion Style, 2009.


I haven't read the more recent papers, but the 2009 papers should give you a good sense of how temporal data can be modeled using RBMs and DBNs.
A: I also agree with Ran, most of the deep learning techniques are tested with image data set. Please checkout this research paper, it talks about audio classification using deep learning techniques.  
A: I can't post comments yet, but I have the following remark.
With multivariate data, I usually think of different "types" of data, e.g. a mixture of bool, multiselection, or floating point data (here it is called mixed-variate). As I see it, the input of different audio signals is therefore not multivariate, but multidimensional, because you probably have same data type (real valued data) for all channels.
I think most of the basic RNN can handle highdimensional time series inbut. See e.g. here.
Maybe a combination of the two above links will lead to a reasonable algorithm.
