1
$\begingroup$

Recently, I dug into RNN and LSTM and, if I understood well, the memory is passing through the network.

So I wonder why in some architectures, the number of cells in the first LSTM layer is higher than the length sequence ? Would that mean that the "exceeding cells" (w.r.t. to the sequence length) have no other input than the memory of the previous cell (but no "data" input) ? According to me, the only effect of adding cells (or a second layer) would thus be to increase non-linearity and "combinations" but there is no improvement in terms of memory and "temporal knowledge"...

Am I right ?

Improve this question
$\endgroup$
1
  • $\begingroup$ can you provide a reference to such an architecture which has more cells than inputs? $\endgroup$
    – shimao
    Commented Jul 22, 2018 at 17:16

1 Answer 1

Reset to default
4
+50
$\begingroup$

All cells will have an input of some kind.

If the LSTM continues iterating after the sequence ends, one of two circumstances typically obtains:

  1. The missing inputs are replaced by padding (usually 0s)
  2. The cell begins receiving input from another source. This may be the output of the previous cell, the output of of cells at a deeper layer, or some appropriately shaped vector/tensor computed over the entire sequence length (eg, as maybe computed by some attention mechanism). Often it is taken from the final output of the model at the previous time step.

I think your intuition about the number of cells is correct, in that just inputting zero pads (what I think you mean by "no data") and letting the cell run on would mostly result in the degradation of the memory state. However, I'm not aware of any typical architectures that do this. Usually the output is being fed back in as input in this circumstance.

However, adding additional layers adds additional memory blocks. The effect of this is complicated, but it does increase the total size of memory available to the model. There's no guarantee that this will improve the temporal depth of memory, but in principle it could have that effect. (Relatedly, some researchers have even tried used "hierarchical" LSTMs to take advantage of deeper layers specifically for increasing temporal depth).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.