1
$\begingroup$

Existing research documents LSTMs to perform poorly with timesteps > 1000 - i.e., inability to "remember" longer sequences. What's absent explicit mention is whether this applies for one or more of the following:

  • Many-to-Many - return $t$ outputs for $t$ input timesteps, as with Keras' return_sequences=True
  • Many-to-One - return only the last output for $t$ input timesteps (return_sequences=False)
  • Both, stacked - with =True preceding =False

In 'both', it's unclear whether the former's input (and thus output, i.e. input to latter) should be limited to <1000 timesteps, or that it transforms input timesteps in some manner that effectively 'lightens the load' on latter's memory. For one, Keras' =False LSTM utilizes more than double the =True's number of weights; I don't know why it does so, but it does imply greater model capacity.

So, does Many-to-Many, or Many-to-Many stacked with Many-to-One LSTM - bear greater memory capacity? Experimental or theoretical insights appreciated.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Model capacity is mostly orthogonal to long term dependency learning -- you can have a huge model with very high capacity, but which doesn't learn long term dependencies well. You could also theoretically have a model which remembers long term dependencies very well, but has very limited capacity.

So no, I don't think stacking additional LSTM layers would help here. Also many-to-one and many-to-many aren't really directly comparable.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the info; by "capacity" I do intend only the long-term memory capacity. If neither stacking nor greater cell size necessarily aids learning long-term dependencies, then what does? Surely a 50-cell LSTM, in every sense of the word, has greater capacity than a 1-cell LSTM for a large dataset. Is trial & error the only way? $\endgroup$
    – OverLordGoldDragon
    Commented Jul 18, 2019 at 17:54
  • $\begingroup$ There is a distinction between 'model capacity' and 'memory capacity'. The former, used in the answer, refers to the size of the network. If the model capacity is very large for a particular dataset, the model tends to overfit (i.e. learn to reproduce the sequences in the training set) and therefore doesn't learn LT dependencies well. Note that the correct model capacity thus depends on both the size and variability of the data, such that trial & error tends to be necessary. $\endgroup$
    – GR4
    Commented Jul 19, 2019 at 10:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.