Interconnections between embeddings layer and LSTM layer

Question

I'm trying to build a text classifier with keras using word embeddings (glove) and a RNN (in this case a LSTM) using keras. I searched in several sites and decided to start with this configuration:

model = Sequential()
model.add(embeddingLayer)
model.add(LSTM(LSTM_DIM, dropout=DROPOUT))
model.add(Dense(NUM_CLASSES, activation='sigmoid'))

rmsprop = optimizers.RMSprop(lr=LEARNING_RATE)
model.compile(loss='categorical_crossentropy',
              optimizer=rmsprop,
              metrics=['acc'])

The embeddings Layer is a 60693x300 matrix being the first number the vocabulary size of my training set and 300 the embedding dimension. The input vectors are limited to 100 words, so when I multiply them to the embeddings matrix I get a 100x300 matrix being each row the embedding of the word present in the input. I also understand that each LSTM cell is connected to the neurons in the output layer (I have four classes so NUM_CLASSES=4), also LSTM_DIM is set to 50. However I can't figure out the connections between the input layer and the hidden (LSTM) layer.

When I execute model.summary() I get the following result:

Model: "sequential"

Layer (type)	Output Shape	Param #
embedding (Embedding)	(None, 100, 300)	18207900
lstm (LSTM)	(None, 50)	70200
dense (Dense)	(None, 4)	204

Total params: 18,278,304 Trainable params: 70,404 Non-trainable params: 18,207,900

I need to understand how the 100x300 input matrix is fed into the LSTM layer. Is it one word (1X300) to each LSTM cell at the time?

I would appreciate if someone could explain me which are the interconnections and how is this semantically interpreted.

Thanks in advance

Answer 1

LSTM (as well as all recurrent neural networks) process one input vector, in this case corresponding to one word, at a time.

In each step, the RNN cell reads the input vector and updates its hidden and output states accordingly (LSTM distinguishes hidden state $c_t$ and output state $h_t$, other RNN types typically don't). Depending on the attributes of the LSTM calls, it either returns one state vector (if return_sequences is set to True) per input or the last hidden after reading the entire input sequence. In this code snippet, it is the last output state of dimension 50.