Clarification on Number of LSTM cells per layer

Question

When the Seq2Seq paper by Sutskever et al. refers to building an LSTM with a k-dimensional cell size, does this refer to both the feature vector size of vectors $C_t$ (cell), and $h_t$ (hidden state) in the LSTM? (Page 5; Top). The paper later says that the feature vector dimensionality of the word embeddings that they use is 1000 dim (just as the number of LSTM cells).

Isn't this a bit redundant, since the Embedding is an input to the LSTM cells, and the $C_t$ and $h_t$ vectors should both have the same size given the LSTM equation definition:

$h_t = o_t \odot tanh(C_t) $, where $\odot$ is the element-wise multiplication

Or can the cell size and hidden state size actually be different?

Answer 1

• the cell dimension and hidden state dimension should be the same, for the reason you've given $(h_t = o_t \odot tanh(C_t)) $.
• the word embedding dimension does not have to be the same as the hidden state dimension.