1
$\begingroup$

I understand that in LSTM, there are 3 gates to help keep the memory. But why do we use LSTM instead of NN in the first place? For example, in a LSTM, my features are just 1-D, time steps is set to be 60, which means the longest memory my LSTM can remember is just 60 days ago, you can't use any information 73 days ago to predict today's target.

But in a plain NN, we can reconstruct the 1-D variable into a 60-D variables with the other dimensions being the variable's value 1-day ago, 2-day ago, ..., 60-day ago. And we predict today's target variable.

What is the difference here? We are all just using 60-day information to predict a single variable.

Improve this question
$\endgroup$
0

2 Answers 2

Reset to default
2
$\begingroup$

A couple of reasons:

First, you might not know how long your sequence will be ahead of time. RNNs let you use a variable length input.

Second, RNNs are a good way to do parameter sharing between different time steps. Instead of having to learn how to extract useful information from each time step separately you can reuse the same parameter matrix recurrently.

Improve this answer
$\endgroup$
2
  • $\begingroup$ What do you mean by 'Rnns let you use a variable input length input'? In the LSTM in keras, there is a parameter input_shape=(batch_size, time_steps, n_features). Do you mean by picking different time_steps each trial? And check the results on validation set to pick the best time_steps? $\endgroup$
    – StayFoolish
    Commented Sep 28, 2017 at 7:10
  • $\begingroup$ The time_steps that you are specifying there is the maximum number that you expect to see. Not all the sequences need to be the same length. We use LSTMs a lot in speech recognition and translation. Each sentence is a different length and the LSTM handles it perfectly. $\endgroup$
    – Aaron
    Commented Sep 28, 2017 at 17:36
1
$\begingroup$

First to clarify: There are 3 types of NNets used for this type of problem. Feed forward NNets (i.e plain NN), Recurrent NNets (RNN), and LSTM, which is a more advanced class of RNN.

The problem with plain NN is that they don't have any memory. Let's take your example of:

But in a plain NN, we can reconstruct the 1-D variable into a 60-D variables with the other dimensions being the variable's value 1-day ago, 2-day ago, ..., 60-day ago. And we predict today's target variable.

Now say there are recursive relationships between you data points, so that the Data(Today) = f[Data(Yesterday)] = f[f[Data(2 Days ago)]], etc...

A plain NN can learn the relationship between these data points during the training phase, but once the training is done, it has no way of applying such recursive relationships for future data points because the data between the inputs and the outputs flows in only one direction.

Putting it another way, a plain NN can predict the value for today. But if you wanted to predict the target variable for 2 days, 3 days, or N days ahead (see one-step ahead vs. multi-sept forecasting) - then the NN has no way of predicting those values, since it has no way of feeding the forecast for today back as an input to generate a forecast for tomorrow and the day after.

RNN and LSTM on the other hand, have feedback loops going from the outputs back to the inputs, which allow them to perform multistep forecasting.

Moreover, LSTM have additional properties that allow them to learn long term dependencies, whereas regular RNN can only learn short dependencies.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.