Multivariate time series are, to the best of my understanding, one of the few cases where Deep Learning still hasn't had its AlexNet moment. I'm especially interested to the case where most of the time series are continuous variables, with a few being categorical variables. To fix the ideas, think sensors of a big industrial machine - some sensors will record real-valued time series (pressure, temperature, speed, etc.) and some other categorical time series ("running/non running", "alarm 1/alarm 2/ no alarm", valve open/ valve closed", etc.). If the introduction of categorical variables makes the problem too hard, no issues - we can consider only continuous variables.

Some of these time series have a lot of missing data (the norm, rather than the exception, in industry) and I'd like to perform missing value imputation. In theory, a generative model seems the perfect fit for such a problem, thus I thought of VAEs. In practice, all the applications of VAE to missing data imputation that I know of, are related to images. Never seen them applied to missing data imputation for multivariate times series.

Can they be used? If so, which is the major modification I need to make to the architecture, in order to get a performing implementation for my use case? If no, which other Deep Learning or AI model are suitable for missing data imputation with multivariate time series?

  • 1
    $\begingroup$ I'm curious: are there any examples of deep learning doing well in an area of application in which there is a decent amount of noise in the response? By noise in the response, I mean that even a perfect model will have a good deal of uncertainty. To illustrate, classification in images generally does not have noise in the response; a picture of a dog is usually unquestionably a picture of a dog, although generating a rule to go from pixel values to classification is likely complicated. Calling a coin flip with only mild bias would have lots of noise; the best model can't do better than $p$. $\endgroup$ – Cliff AB Mar 24 '19 at 1:48
  • $\begingroup$ We have a recent paper on this problem: arxiv.org/abs/1907.04155 $\endgroup$ – Vincent Jul 11 '19 at 12:05
  • $\begingroup$ @Vincent then write an answer here $\endgroup$ – DeltaIV Jul 11 '19 at 15:58

Wouldn't you consider language to be a type of time series? How about OpenAI Five's representation of the Dota game state as a time series with 20,000 continuous and discrete variables?

The tool of choice for such sequence modeling is LSTMs, Transformers, and other autoregressive models. And you can always tack on a latent prior to any of these models, (see "recurrent VAE"), but it's not necessary because any sequence distribution can be modeled as $p(x_0, x_1, \ldots) = p(x_0) \prod_j p(x_j|x_{i<j})$.

The difficulty with imputation using these models is that you'd presumably be trying to do something like $\max_{X_i \ldots X_j} p(x_0 = X_0, \ldots x_{i-1}=X_{i-1}, x_i=X_i, \ldots, x_j=X_j, \ldots)$, and due to the autoregressive nature of the model, there's actually no easy way to carry out the maximization. If you just want to sample $X_i \ldots X_j$ from the modeled distribution with the other values fixed, there's no easy way to do that either. Maybe it would not be too expensive to use some form of MCMC sampling.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.