I'm trying to build a text classifier with keras using word embeddings (glove) and a RNN (in this case a LSTM) using keras. I searched in several sites and decided to start with this configuration:

model = Sequential()
model.add(LSTM(LSTM_DIM, dropout=DROPOUT))
model.add(Dense(NUM_CLASSES, activation='sigmoid'))

rmsprop = optimizers.RMSprop(lr=LEARNING_RATE)

The embeddings Layer is a 60693x300 matrix being the first number the vocabulary size of my training set and 300 the embedding dimension. The input vectors are limited to 100 words, so when I multiply them to the embeddings matrix I get a 100x300 matrix being each row the embedding of the word present in the input. I also understand that each LSTM cell is connected to the neurons in the output layer (I have four classes so NUM_CLASSES=4), also LSTM_DIM is set to 50. However I can't figure out the connections between the input layer and the hidden (LSTM) layer.

When I execute model.summary() I get the following result:

Model: "sequential"

Layer (type) Output Shape Param #
embedding (Embedding) (None, 100, 300) 18207900
lstm (LSTM) (None, 50) 70200
dense (Dense) (None, 4) 204

Total params: 18,278,304 Trainable params: 70,404 Non-trainable params: 18,207,900

I need to understand how the 100x300 input matrix is fed into the LSTM layer. Is it one word (1X300) to each LSTM cell at the time?

I would appreciate if someone could explain me which are the interconnections and how is this semantically interpreted.

Thanks in advance


1 Answer 1


LSTM (as well as all recurrent neural networks) process one input vector, in this case corresponding to one word, at a time.

In each step, the RNN cell reads the input vector and updates its hidden and output states accordingly (LSTM distinguishes hidden state $c_t$ and output state $h_t$, other RNN types typically don't). Depending on the attributes of the LSTM calls, it either returns one state vector (if return_sequences is set to True) per input or the last hidden after reading the entire input sequence. In this code snippet, it is the last output state of dimension 50.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.