0
$\begingroup$

I want to predict a multivariate time series. My time series is $a_1(t),...,a_{k-1}(t)$ and I want to predict $a_k(t)$. I use the following keras LSTM:

model = Sequential()
model.add(LSTM(90,return_sequences=True,input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(LSTM(90))
model.add(Dense(1))

I use $a_1(t),...,a_{k-1}(t)$ as input and $a_k(t)$ as output to train it. So I don't use delayed inputs like $a_s(t-l)$. My qestion is in this situation, Does LSTM work as a deep neural network? i.e. Is it same as the following keras net?

model.add(Dense(90, input_dim=12))
model.add(Dense(90))
model.add(Dense(1))
$\endgroup$

1 Answer 1

1
$\begingroup$

Welcome to Cross-Validated!!

Two comments:

First, you seem to be mixing the concept of depth of a neural network with the concept of recursion in a RNN or LSTM cell. Although for the case of LSTM they can seem like very similar or overlapping concepts, they are in fact not.

  • The depth of a network, regardless of whether it uses dense neurons, or LSTM blocks, or CNN, is the number of hidden layers it has. If a network has at most one hidden layer, then it is a shallow network. If it has two or more hidden layers, it is a deep neural network.
  • The recursiveness of LSTM (and other RNN models in general): An RNN block feeds its output back to its input. Because of this, an RNN or LSTM cell can be represented in one of two ways: As a single neuron with a feedback loop, or as a sequence of neurons without feedback loops. The number of delayed/lagged inputs corresponds to the number of times you run the feedback loop for a given input (note that you can that it can also be infinite - although I'm not familiar with how those work exactly). These illustrations from Chris Olah and Andrej Karpathy are very helpful.

enter image description here enter image description here

The different types of sequential relationships possible with RNN (For example having many delayed inputs and one output is many-to-one, etc...):

enter image description here

Now because conceptually unrolling an RNN neuron into multiple feedforward (i.e. dense) neurons makes it look like we add layers, it is easy to think that the number of steps delays somehow corresponds to additional layers that make the network deep instead of shallow. However, there is a key difference: No matter how many times you unroll an RNN/LSTM cell, it will still have the same set of input weights and biases after training. Successive simple/feedforward neurons in a DNN on the other hand can have different weights and biases, if the training process optimizes their parameters accordingly.

Now you can have an LSTM Network that has multiple input and/or output steps and also multiple hidden layers. Even if there are no delayed inputs or sequential outputs, if there are more than one LSTM hidden layer, than it is still a deep network.

So to answer your main question:

Does LSTM without delayed inputs work as a deep net?

Whether the LSTM model has delayed inputs or not is irrelevant. If it has $ N_{layers} \geq 2 $ then it is a deep net, otherwise no it isn't.

Second (and more important), your problem as you describe it - $a_1(t),...,a_{k-1}(t)$ - as input and $a_k(t)$ as output - is not really a time series problem. First of all, it is much clearer if you use the following notation:

  • Inputs: $x_1(t),x_2(t),...,x_k(t)$
  • Output: $\hat{y}(t)$ (It is customary to use the "^" on top of a predicted value to distinguish it from the real value, so the prediction is $\hat{y}(t)$, and the real observed value is $y(t)$).

Now having cleared the notation, lets look at your model: You are trying to use an LSTM (or some other DL architecture) to fit something that looks like: $$\hat{y}(t) = f(x_1(t),x_2(t),...,x_k(t))$$ A time series model on the other hand will have the form: $$\hat{y}(t+1) = f(y(t),y(t-1),...,y(t-k))$$ Or $$\hat{y}(t+1) = f(x(t),x(t-1),...,x(t-k))$$

In your case, you have two options:

  • If you expect the relationship $\hat{y}(t) = f(x_1(t),x_2(t),...,x_k(t))$ to not change over time, then just use regression (linear, or non linear like XGRegressor). If you have your heart set on Deep Learning, just use DNN or Deep and Wide. Don't use LSTM.
  • On the other hand, if you have a situation where the relationship $\hat{y}(t) = f(x_1(t),x_2(t),...,x_k(t))$ is time varying, then that is a much more complicated use case. It would be a case of time varying regression or Dynamic Linear Models.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.