1
$\begingroup$

I'm trying to design a DNN for time series prediction. I have time series data with 2 features. I would like to leverage on stacked LSTM layers due to it's powerful predicting capability. However I'm not sure how to combine the 2 features and still use stacked LSTM model.
I think the first layer is 2 separate LSTMs, one for feature1 and one for feature2. I would like to merge the 2 output somehow and feed it to the next layer, the stacked LSTM. So it would be something like this: 

Input1; Input2
LSTM_1(Feature1); LSTM_2(Feature2)
Combine LSTM_1 and LSTM_2 output
LSTM_3

I have the following questions:
In general do I think in the right direction to use both features with LSTM?
What is the common way for the step of combining, if any? Average? Sum?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

An LSTM layer can combine multiple inputs. From this perspective it is not different than ordinary neural network layers.

Ordinary neural network layers consists of multiple units. Each of these units gets input from all the (weighted) activations of the previous layer. Likewise, an LSTM unit also gets input from the (weighted) activations of the previous layer, in your case the two inputs.

An LSTM unit is different in that it does some fancy stuff (the forget and output gates) with the input and that it has additinal recurrent connections. But that does not change the fact that they have arbitrarily many inputs from the previous layer.

Your network could thus look like this: 

Inputs (input1; input2; etc.)
LSTM 1 (receiving actions from all inputs)
etc.
Improve this answer
$\endgroup$
4
  • $\begingroup$ Also see this blogpost by Christopher Olah: colah.github.io/posts/2015-08-Understanding-LSTMs $\endgroup$
    – Pieter
    Commented Aug 28, 2017 at 15:37
  • $\begingroup$ Yes, I can feed 2 features to certain LSTM implementations (but not to the base model from '97). My tools (tensorflow) seem to combine the inputs before the recurrent calculations take place because adding a new feature only slightly increases the number of parameters in the network. as a result the additional feature doesn't improve accuracy at all however training a network only with feature2 proves that it's useful. $\endgroup$
    – Manngo
    Commented Aug 28, 2017 at 16:00
  • $\begingroup$ Also take a look at: keras.io $\endgroup$
    – Pieter
    Commented Aug 28, 2017 at 16:01
  • $\begingroup$ Yes, this is exactly what I'm using. $\endgroup$
    – Manngo
    Commented Aug 28, 2017 at 16:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.