I'm looking for a way to model and extract features from multivariate temporal data (e.g., multi-channel audio recordings).

I'm specifically interested in deep learning methods such as RBM, sparse autoencoders and so on.

Most methods I encountered consider only one dimension of the data, or maybe a 2D "block" in the data (usually images).

I couldn't find a paper on how to take into consideration the temporal aspect and also the multivariate aspect. For example if I'm recording audio from multiple channels, then I know that the channels are probably correlated to some degree. Also, since it's an evolving temporal signal, there is also some correlation of each sample to the previous samples.

Is there some way to incorporate these multivariate temporal constraints into a network?

I'm looking for specific papers on methods for multivariate temporal where the multi-channel structure is used.



3 Answers 3


I also agree with Ran, most of the deep learning techniques are tested with image data set. Please checkout this research paper, it talks about audio classification using deep learning techniques.

  • $\begingroup$ I actually know this paper. They are converting their 1D temporal signal to a 2D frequency X time matrix with a spectrogram and the used 1D "filters" for the connections to the hidden units. It doesn't seem like they are looking at the frequency-time structure. Am I wrong? $\endgroup$
    – Ran
    Feb 15, 2012 at 15:15

There are multiple papers by Hinton et al. which deal with temporal data and also audio (http://www.cs.toronto.edu/~hinton/papers.html). For example:

  • Acoustic Modeling using Deep Belief Networks, 2012.
  • Learning a better Representation of Speech Sound Waves using Restricted Boltzmann Machines, 2011.
  • Deep Belief Networks using Discriminative Features for Phone Recognition, 2011. The Recurrent Temporal Restricted Boltzmann Machine, 2009.
  • Factored Conditional Restricted Boltzmann Machines for Modeling Motion Style, 2009.

I haven't read the more recent papers, but the 2009 papers should give you a good sense of how temporal data can be modeled using RBMs and DBNs.

  • 1
    $\begingroup$ My question was specific for multivariate temporal data, not a single channel temporal data (like those papers). thanks. $\endgroup$
    – Ran
    Feb 16, 2012 at 9:46
  • 3
    $\begingroup$ I probably misunderstand your use of the term multivariate, but as I see it these papers do model multivariate temporal data. For example, the model described in the last paper was used to model motion capture data, where at each point in time you have multiple measurements. Unless you can make some specific independence assumptions about your channels, I don't see why couldn't model them in the same manner. Please clarify. $\endgroup$
    – Lucas
    Feb 16, 2012 at 10:02

I can't post comments yet, but I have the following remark.

With multivariate data, I usually think of different "types" of data, e.g. a mixture of bool, multiselection, or floating point data (here it is called mixed-variate). As I see it, the input of different audio signals is therefore not multivariate, but multidimensional, because you probably have same data type (real valued data) for all channels.

I think most of the basic RNN can handle highdimensional time series inbut. See e.g. here.

Maybe a combination of the two above links will lead to a reasonable algorithm.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.