Stateful LSTM internal memory size

Question

I read Understanding LSTM Networks and I'm trying to understand the internal state of LSTM (C_t). According to Stateful LSTM in Keras (paragraph Mastering stateful models), sequence elements can be fed to a stateful LSTM network one by one (without sliding window).

Let's say I expect my sequences to have length of 50 or less. How do I define my LSTM to have enough internal memory to remember sequences of that length?

As I understand it:

• [Internal memory / internal state / C_t] is a vector whose length is the same as the depth of the recurrence in the network (the number of green boxes in the image below).
• It has nothing to do with the units parameter of the network (which is the number of output features)

(picture from here)

I ran a simple experiment to observe C_t:

from keras.models import Model
from keras.layers import Input
from keras.layers import LSTM
import numpy as np


batch_size = 1
timesteps = 1
input_features_count = 1
output_features_count = 1
inputs1 = Input(batch_shape=(batch_size, timesteps, input_features_count))
lstm1 = LSTM(units = output_features_count, return_sequences=True, return_state=True, stateful=True)(inputs1)
model = Model(inputs=inputs1, outputs=lstm1)
data = array([0.1]).reshape((batch_size, timesteps, input_features_count))
pred_seq, state_h, state_c = model.predict(data)
print(pred_seq.shape) # (1, 1, 1)
print(state_h.shape) # (1, 1)
print(state_c.shape) # (1, 1)


batch_size = 1
timesteps = 2
input_features_count = 3
output_features_count = 5
inputs1 = Input(batch_shape=(batch_size, timesteps, input_features_count))
lstm1 = LSTM(units = output_features_count, return_sequences=True, return_state=True, stateful=True)(inputs1)
model = Model(inputs=inputs1, outputs=lstm1)
data = array([1,2,3,4,5,6]).reshape((batch_size, timesteps, input_features_count))
pred_seq, state_h, state_c = model.predict(data)
print(pred_seq.shape) # (1, 2, 5)
print(state_h.shape) # (1, 5)
print(state_c.shape) # (1, 5)

And now I'm lost, because C_t's length is always 1 (event with sliding window: timesteps=2), which as I see it, is not enough memory to learn any sequence. I also cannot find a way to arbitrarily define internal memory size.

What am I missing here?

Answer 1

As I understand it: [Internal memory / internal state / C_t] is a vector whose length is the same as the depth of the recurrence in the network (the number of green boxes in the image below).

Your understanding is flawed. The RNN cell's internal memory is not the same length as the number of time steps. Instead, at each time step, the memory of the cell is updated. The values of the cell change to reflect the model's understanding of what is "important to remember" given the model weights, its memory state from the previous time-step, and the input.

How do I define my LSTM to have enough internal memory to remember sequences of that length?

This isn't how LSTMs work. Consider a very simple sequence, one that starts at 0 and adds 1 at each time step. Now I tell you that the value at $f(10)=10$. Do you need to know $f(9)=9, f(8)=8, \cdots, f(0)=0$ to predict $f(11)$? Of course not. LSTMs generalize this idea to more complex sequences by learning to approximate how the values change between time steps, and then applying that rule to make predictions.

Is internal state a vector at all?

print(state_h.shape) # (1, 5)
print(state_c.shape) # (1, 5)

It looks like each is a vector!

What determines it's length?

output_features_count = 5
lstm1 = LSTM(units = output_features_count, ...)

Let's consider an example with a little more complex sub-sequences: [1,0,0,0]->1, [1,0,0,1]->2, [0,0,0,1]->3. Now in order to make a prediction (after the 4th input) we kind of have to remember the first and the second inputs (third is meaningless). I thought this information was stored in the internal state. Is that correct?

It uses the memory cells to represent the information from the first 3 states. It doesn't need to explicitly store that information in the "raw" form of the original sequences.