# Stateful LSTM internal memory size

I read Understanding LSTM Networks and I'm trying to understand the internal state of LSTM (C_t). According to Stateful LSTM in Keras (paragraph Mastering stateful models), sequence elements can be fed to a stateful LSTM network one by one (without sliding window).

Let's say I expect my sequences to have length of 50 or less. How do I define my LSTM to have enough internal memory to remember sequences of that length?

As I understand it:

• [Internal memory / internal state / C_t] is a vector whose length is the same as the depth of the recurrence in the network (the number of green boxes in the image below).
• It has nothing to do with the units parameter of the network (which is the number of output features)

(picture from here)

I ran a simple experiment to observe C_t:

from keras.models import Model
from keras.layers import Input
from keras.layers import LSTM
import numpy as np

batch_size = 1
timesteps = 1
input_features_count = 1
output_features_count = 1
inputs1 = Input(batch_shape=(batch_size, timesteps, input_features_count))
lstm1 = LSTM(units = output_features_count, return_sequences=True, return_state=True, stateful=True)(inputs1)
model = Model(inputs=inputs1, outputs=lstm1)
data = array([0.1]).reshape((batch_size, timesteps, input_features_count))
pred_seq, state_h, state_c = model.predict(data)
print(pred_seq.shape) # (1, 1, 1)
print(state_h.shape) # (1, 1)
print(state_c.shape) # (1, 1)

batch_size = 1
timesteps = 2
input_features_count = 3
output_features_count = 5
inputs1 = Input(batch_shape=(batch_size, timesteps, input_features_count))
lstm1 = LSTM(units = output_features_count, return_sequences=True, return_state=True, stateful=True)(inputs1)
model = Model(inputs=inputs1, outputs=lstm1)
data = array([1,2,3,4,5,6]).reshape((batch_size, timesteps, input_features_count))
pred_seq, state_h, state_c = model.predict(data)
print(pred_seq.shape) # (1, 2, 5)
print(state_h.shape) # (1, 5)
print(state_c.shape) # (1, 5)


And now I'm lost, because C_t's length is always 1 (event with sliding window: timesteps=2), which as I see it, is not enough memory to learn any sequence. I also cannot find a way to arbitrarily define internal memory size.

What am I missing here?

As I understand it: [Internal memory / internal state / C_t] is a vector whose length is the same as the depth of the recurrence in the network (the number of green boxes in the image below).

Your understanding is flawed. The RNN cell's internal memory is not the same length as the number of time steps. Instead, at each time step, the memory of the cell is updated. The values of the cell change to reflect the model's understanding of what is "important to remember" given the model weights, its memory state from the previous time-step, and the input.

How do I define my LSTM to have enough internal memory to remember sequences of that length?

This isn't how LSTMs work. Consider a very simple sequence, one that starts at 0 and adds 1 at each time step. Now I tell you that the value at $f(10)=10$. Do you need to know $f(9)=9, f(8)=8, \cdots, f(0)=0$ to predict $f(11)$? Of course not. LSTMs generalize this idea to more complex sequences by learning to approximate how the values change between time steps, and then applying that rule to make predictions.

Is internal state a vector at all?

print(state_h.shape) # (1, 5)
print(state_c.shape) # (1, 5)


It looks like each is a vector!

What determines it's length?

output_features_count = 5
lstm1 = LSTM(units = output_features_count, ...)


Let's consider an example with a little more complex sub-sequences: [1,0,0,0]->1, [1,0,0,1]->2, [0,0,0,1]->3. Now in order to make a prediction (after the 4th input) we kind of have to remember the first and the second inputs (third is meaningless). I thought this information was stored in the internal state. Is that correct?

It uses the memory cells to represent the information from the first 3 states. It doesn't need to explicitly store that information in the "raw" form of the original sequences.

• Thanks for the answer, but would you mind clearing some things up? Is internal state a vector at all (according to the first link in my question it should be)? What determines it's length? Let's consider an example with a little more complex sub-sequences: [1,0,0,0]->1, [1,0,0,1]->2, [0,0,0,1]->3. Now in order to make a prediction (after the 4th input) we kind of have to remember the first and the second inputs (third is meaningless). I thought this information was stored in the internal state. Is that correct? – Andrzej Gis Jun 24 '18 at 23:06
• @gisek I've updated my answer. – Sycorax says Reinstate Monica Jun 24 '18 at 23:34
• Ok, I think I get it now. Keras documentation is a little misleading: "units: Positive integer, dimensionality of the output space." I understood is as the number of features to predict, while it seems to be the length of internal state (some abstract representation of changes in a sequence). Thanks a lot for the clarification. – Andrzej Gis Jun 24 '18 at 23:45
• I've only ever found the Stan documentation to be any good. A typical LSTM structure is [input] -> [LSTM units] -> [fully-connected layer(s)] -> [output]. So even if you just have 1 output, it could be helpful to have a number of LSTM units used simultaneously. – Sycorax says Reinstate Monica Jun 24 '18 at 23:47
• Yes, I've seen such models a lot in various examples. I'm wondering though, why not replace the fully-connected (Dense) layer with LSTM(units=1, return_sequences=false)? Does it make any difference? – Andrzej Gis Jun 24 '18 at 23:52