1
$\begingroup$

When the Seq2Seq paper by Sutskever et al. refers to building an LSTM with a k-dimensional cell size, does this refer to both the feature vector size of vectors $C_t$ (cell), and $h_t$ (hidden state) in the LSTM? (Page 5; Top). The paper later says that the feature vector dimensionality of the word embeddings that they use is 1000 dim (just as the number of LSTM cells).

Isn't this a bit redundant, since the Embedding is an input to the LSTM cells, and the $C_t$ and $h_t$ vectors should both have the same size given the LSTM equation definition:

$h_t = o_t \odot tanh(C_t) $, where $\odot$ is the element-wise multiplication

Or can the cell size and hidden state size actually be different?

$\endgroup$

1 Answer 1

1
$\begingroup$
  • the cell dimension and hidden state dimension should be the same, for the reason you've given $(h_t = o_t \odot tanh(C_t)) $.
  • the word embedding dimension does not have to be the same as the hidden state dimension.
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.