Multilayer neural networks for multivariate temporal data

Question

I'm looking for a way to model and extract features from multivariate temporal data (e.g., multi-channel audio recordings).

I'm specifically interested in deep learning methods such as RBM, sparse autoencoders and so on.

Most methods I encountered consider only one dimension of the data, or maybe a 2D "block" in the data (usually images).

I couldn't find a paper on how to take into consideration the temporal aspect and also the multivariate aspect. For example if I'm recording audio from multiple channels, then I know that the channels are probably correlated to some degree. Also, since it's an evolving temporal signal, there is also some correlation of each sample to the previous samples.

Is there some way to incorporate these multivariate temporal constraints into a network?

I'm looking for specific papers on methods for multivariate temporal where the multi-channel structure is used.

Thanks.

Answer 1

I also agree with Ran, most of the deep learning techniques are tested with image data set. Please checkout this research paper, it talks about audio classification using deep learning techniques.

Answer 2

There are multiple papers by Hinton et al. which deal with temporal data and also audio (http://www.cs.toronto.edu/~hinton/papers.html). For example:

• Acoustic Modeling using Deep Belief Networks, 2012.
• Learning a better Representation of Speech Sound Waves using Restricted Boltzmann Machines, 2011.
• Deep Belief Networks using Discriminative Features for Phone Recognition, 2011. The Recurrent Temporal Restricted Boltzmann Machine, 2009.
• Factored Conditional Restricted Boltzmann Machines for Modeling Motion Style, 2009.

I haven't read the more recent papers, but the 2009 papers should give you a good sense of how temporal data can be modeled using RBMs and DBNs.