8
$\begingroup$

RNNs are remarkably good for capturing the time-dependence of sequential data. However, what happens when the sequence elements aren't equally spaced in time?

E.g., the first input to the LSTM cell happens on Monday, then no data from Tuesday to Thursday, and then finally new inputs for each of Friday, Saturday, Sunday. One possibility would be to have some kind of NULL vector being fed for Tuesday through Thursday, but that seems to be a silly solution, both because the NULL entries will contaminate the data and because it's a waste of resources.

Any ideas? How do RNNs handle such cases? If there are methods other than RNNs, I welcome those suggestions as well.

$\endgroup$
  • $\begingroup$ What kind of data is it? Like can we assume that on some days we get a measurement of some feature that's continuous? For example, the temperature at some location? $\endgroup$ – user0 Apr 1 '18 at 19:25
  • $\begingroup$ @user99889 Yes, that could be an example, but I was more thinking of a counter. For example, you get one count at 9:32 am, then nothing, them two counts at 9:37 am, then nothing, etc... $\endgroup$ – Tfovid May 2 '18 at 11:14
2
$\begingroup$

If you are feeding in some data vector $v_t$ at time $t$, the straightforward solution is to obtain a one-hot encoding of the day of week, $d_t$, and then simply feed into the network the concatenation of $v_t$ and $d_t$. The time/date encoding scheme can be more complicated if the time format is more complicated than just day of week of course.

Also, depending on exactly how sparse and irregular the data is, NULL entries should be a reasonable solution. I suspect that the input gate of an LSTM would allow the LSTM to properly read off the information of a NULL entry without contaminating the data (the memory/hidden state) as you put it.

$\endgroup$
  • 1
    $\begingroup$ Actually, instead of one-hot encoding, why not just concatenate a single input node to the LSTM which contains the date, or better yet, the time before the "present", where the present is the last element in the sequence. In this case, those nodes would contain -5 (for Monday), -2 (for Friday), -1 (for Saturday), 0 (for Sunday, i.e., the most recent day). This will circumvent the need for NULLs and at the same time (hopefully) encode by how much the LSTM should "forget" between the recursions. Does this sound reasonable? $\endgroup$ – Tfovid Nov 8 '17 at 15:28
  • $\begingroup$ Yes, that should work. $\endgroup$ – shimao Nov 10 '17 at 16:27
1
$\begingroup$

I would try incorporating time interval explicitly into the model. For instance, a conventional time series models such as autoregressive AR(p) can be thought of as discretizations of continuous time model. For instance, AR(1) model: $$y_t=c+\phi y_{t-1}+\varepsilon_t$$ can be thought of as a version of: $$y_t=c\Delta t+e^{-\gamma\Delta t}y_{t-\Delta t}+\xi_t\sigma\sqrt {\Delta t}$$

You could draw analogies to time series models from RNN. For instance, $\phi$ in AR(1) process can be seen as a memory weight in RNNs. Hence, you could plug the time difference between observations into your features this way. I must warn that it's just an idea, and I didn't try it myself yet.

$\endgroup$
1
$\begingroup$

I think it depends on the data. For example, if you are processing counts and you just forgot to measure it on some days, then the best strategy is to impute the missing values (e.g., via interpolation or Gaussian processes) and then process the imputed time series with an RNN. By imputing, you would be embedding knowledge.

If the missingness is meaningful (it was too hot too measure counts on some days), then it's best to impute perhaps and also append an indicator vector that is 1 if the value was missing and 0 otherwise.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.