RNN for irregular time intervals?

Question

RNNs are remarkably good for capturing the time-dependence of sequential data. However, what happens when the sequence elements aren't equally spaced in time?

E.g., the first input to the LSTM cell happens on Monday, then no data from Tuesday to Thursday, and then finally new inputs for each of Friday, Saturday, Sunday. One possibility would be to have some kind of NULL vector being fed for Tuesday through Thursday, but that seems to be a silly solution, both because the NULL entries will contaminate the data and because it's a waste of resources.

Any ideas? How do RNNs handle such cases? If there are methods other than RNNs, I welcome those suggestions as well.

Answer 1

I just wrote a blog post on that topic!

In short, I write about different methods for dealing with the problem of sparse / irregular sequential data.

Here is a short outline of methods to try:

• Lomb-Scargle Periodogram
  This is a way of computing spectrograms on non-equidistant timestep series.

• Data modeling with Interpolation networks
  You really don't want to interpolate naively between timesteps, but training a network to interpolate for you might help!

• Neural Ordinary Differential Equation models
  Neural networks that can work with continuous time can naturally work on irregular time series.

• Add timing dt to the input as an additional feature (or positional encoding in Tensorflow)

• Methods for dealing with missing values
  This is only viable if you have vast amounts of data

Hope this helps point you to the right direction :)

Answer 2

If you are feeding in some data vector $v_t$ at time $t$, the straightforward solution is to obtain a one-hot encoding of the day of week, $d_t$, and then simply feed into the network the concatenation of $v_t$ and $d_t$. The time/date encoding scheme can be more complicated if the time format is more complicated than just day of week of course.

Also, depending on exactly how sparse and irregular the data is, NULL entries should be a reasonable solution. I suspect that the input gate of an LSTM would allow the LSTM to properly read off the information of a NULL entry without contaminating the data (the memory/hidden state) as you put it.

Answer 3

I would try incorporating time interval explicitly into the model. For instance, a conventional time series models such as autoregressive AR(p) can be thought of as discretizations of continuous time model. For instance, AR(1) model: $$y_t=c+\phi y_{t-1}+\varepsilon_t$$ can be thought of as a version of: $$y_t=c\Delta t+e^{-\gamma\Delta t}y_{t-\Delta t}+\xi_t\sigma\sqrt {\Delta t}$$

You could draw analogies to time series models from RNN. For instance, $\phi$ in AR(1) process can be seen as a memory weight in RNNs. Hence, you could plug the time difference between observations into your features this way. I must warn that it's just an idea, and I didn't try it myself yet.

Answer 4

I think it depends on the data. For example, if you are processing counts and you just forgot to measure it on some days, then the best strategy is to impute the missing values (e.g., via interpolation or Gaussian processes) and then process the imputed time series with an RNN. By imputing, you would be embedding knowledge.

If the missingness is meaningful (it was too hot too measure counts on some days), then it's best to impute perhaps and also append an indicator vector that is 1 if the value was missing and 0 otherwise.