6
$\begingroup$

Does LSTM eliminate the need for input lags?

I believe the answer is yes; however, I've not found it explicitly stated in the papers and searching I have completed.

$\endgroup$
5
  • $\begingroup$ Please only include directly relevant information in the post. (Your feelings about being censored are not helpful in making your question clearer or more precise.) $\endgroup$ Commented Sep 6, 2016 at 17:52
  • $\begingroup$ If you feel your questions have not been well-recieved, please consider consulting our help resources, and meta.stats.stackexchange.com. I would say that this question is a bit brief and could stand to have more information and context in order to make it clear to prospective answerers. $\endgroup$
    – Sycorax
    Commented Sep 6, 2016 at 18:53
  • $\begingroup$ I think this is a legitimate question. I have not come across any paper that compares RNN with traditional time series models, or maybe that's my ignorance of the literature. $\endgroup$
    – horaceT
    Commented Sep 6, 2016 at 23:07
  • $\begingroup$ I reduced my question to a Yes/No to satisfy The Rules which discourage questions requesting opinions or qualitative answers. "What is the best way to...." is generally not permitted although a very legitimate question. It seems like the objective of this site is to build a knowledge base and not assist discussion among researchers. $\endgroup$ Commented Sep 7, 2016 at 19:08
  • 1
    $\begingroup$ Could you define what you mean by input lags? Because I'm not sure why an LSTM would do better than a time series when it comes to predicting future values more than one unit of time forward, especially because there is literally no way of extracting such a prediction from an LSTM, unless you supply an additional input indicating how far in the future you want to predict, or if your LSTM outputs some kind of distribution that's time dependent. $\endgroup$
    – Alex R.
    Commented Sep 26, 2018 at 23:13

2 Answers 2

2
$\begingroup$

I believe it is actually pretty clear what you mean by the term input lags, but I will state explicitly. When doing a regression problem with an LSTM, a input signal $ \mathbf{x} \in \mathbb{R}^{n \times t \times c_1 } $ is used to predict another signal $ \mathbf{y} \in \mathbb{R}^{n \times t \times c_2} $. For simplicity I consider $ c_1 = 1, c_2 = 1 $, and I will take one time series $ x[t] $ , so it is possible to talk about in discrete time series terms.

An input delay is then the choice of $ \tau \in \mathbb{Z}^{0+} $, which will transform the signal like $ x_{delayed}[t] = x[t - \tau] $, and for $ t < 0 $, we define $ x[t] = 0 $, so the signal is zero-padded at the beginning, but this is merely a choice of signal processing. With similiar reasoning it is possible to define the concept of output lag too.

In case of LSTMs input lags is typically less concern than output lags in my experience. This could be checked by considering the problem of training an LSTM to predict the delayed version of itself. Consider the LSTM model equations,

$$ f_t = \sigma_g(W_{f} x_t + U_{f} h_{t-1} + b_f) \\ i_t = \sigma_g(W_{i} x_t + U_{i} h_{t-1} + b_i) \\ o_t = \sigma_g(W_{o} x_t + U_{o} h_{t-1} + b_o) \\ c_t = f_t \circ c_{t-1} + i_t \circ \sigma_c(W_{c} x_t + U_{c} h_{t-1} + b_c) \\ h_t = o_t \circ \sigma_h(c_t) $$

The goal is then for the algorithm to learn $ h_t = x_{t-\tau} $. We cannot explicitly learn that, but it is possible to learn weights such that $ c_t = f(x_{t-\tau})$. Considering a mapping of $h_2 = x_1$, we could set $ o_2 = 1 , f_2 = 1, i_1 = 1, i_2 = 0, U_c = 0$, and $ W_c, b_c $ could be chosen to constrain the input values in the approximately linear regime of the sigmoid, so $ h_2 = \sigma_h ( \sigma_c ( x_1 )) \approx x_1 $. An additional regression layer might help to scale back the values from the linear regime of the sigmoid to the original scale. So in terms of the model equations, the parameters exist to circumvent the mapping, learnability is a more involved question to answer, depending on the actual optimisation used.

For output lags, this doesn't eliminate the need however, because an LSTM is a causal model. BLSTM as mentioned above, is acausal, so it might be used to circumvent this problem, however this comes at the cost of sacrificing causality of your model, i.e. real-time signal processing becomes unfeasible.

$\endgroup$
1
  • $\begingroup$ boomkin, Thank you SO MUCH for your effort to answer my question. I confess I am more the physicist that explains things with graphics and not the mathematician that explains with equations, HOWEVER, I will made an effort to claw my way through your post. In my defense, I do have a good intuitive sense of how these methods work. Thanks you again! Tom $\endgroup$ Commented Jan 4, 2019 at 10:12
0
$\begingroup$

No, it doesn't eliminate that need. Sometimes people use a bi-directional LSTM to get information from both sides of a sample before making a prediction. In that case, you wouldn't have to do an input lag.

$\endgroup$
1
  • 6
    $\begingroup$ Can you pls clarify/explain/elaborate how the bi-directional LSTM case relates to OP's question about lag selection. $\endgroup$
    – horaceT
    Commented Sep 26, 2016 at 17:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.