# Difference between a single unit LSTM and 3-unit LSTM neural network

The LSTM in the following Keras code

input_t = Input((4, 1))
output_t = LSTM(1)(input_t)
model = Model(inputs=input_t, outputs=output_t)
print(model.summary())


can be represented as

I understand that when we call model.predict(np.array([[[1],[2],[3],[4]]])) the (only) LSTM unit first processes the vector [1], then [2] plus the feedback from the previous input and so on until the vector [4]. In other words $x_1 = [1], x_2 = [2], x_3 = [3], x_4 = [4]$.

I don't understand how the following neural network would process the same input sequence [1],[2],[3],[4]

 input_t = Input((4, 1))
output_t = LSTM(3)(input_t)
model = Model(inputs=input_t, outputs=output_t)
print(model.summary())


In this NN we have three LSTM units. How do they process the sequence [1],[2],[3],[4]? Are they interconnected with each other? Do they process the input sequence in parallel or does a one unit process the input sequence and later its input is fed to the second LSTM unit and so on? Could someone schematically using this picture explain the whole process?

In Keras LSTM(n) means "create an LSTM layer consisting of LSTM units. The following picture demonstrates what layer and unit (or neuron) are, and the rightmost image shows the internal structure of a single LSTM unit.

The following picture shows how the whole LSTM layer operates.

As we know an LSTM layer processes a sequence, i.e, $\mathbb{x}_1, \dots, \mathbb{x}_N$. At each step $t$ the layer (each neuron) takes the input $\mathbb{x_t}$, output from previous step $\mathbb{h_{t-1}}$, and bias $b$, and outputs a vector $\mathbb{h_t}$. Coordinates of $\mathbb{h_t}$ are outputs of the neurons/units, and hence the size of the vector $\mathbb{h_t}$ is equal to the number of units/neurons. This process continues until $\mathbb{x}_N$.

Now let's compute the number of parameters for LSTM(1) and LSTM(3) and compare it with what Keras shows when we call model.summary().

Let $inp$ be the size of the vector $\mathbb{x_t}$ and $out$ be the size of the vector $\mathbb{h_t}$ (this is also the number of neurons/units). Each neuron/unit takes input vector, output from the previous step, and a bias which makes $inp + out + 1$ parameters (weights). But we have $out$ number of neurons and so we have $out\times(inp + out + 1)$ parameters. Finally each unit has 4 weights (see the rightmost image, yellow boxes) and we have the following formula for the number of parameters: $$4out(inp + out + 1)$$

Let's compare with what Keras outputs.

Example 1.

 t1 = Input(shape=(1, 1))
t2 = LSTM(1)(t1)
model = Model(inputs=t1, outputs=t2)
print(model.summary())

Layer (type)                 Output Shape              Param #
=================================================================
input_2 (InputLayer)         (None, 1, 1)              0
_________________________________________________________________
lstm_2 (LSTM)                (None, 1)                 12
=================================================================
Total params: 12
Trainable params: 12
Non-trainable params: 0
_________________________________________________________________


Number of units is 1, the size of input vector is 1, so $4\times 1 \times (1 + 1 + 1) = 12$.

Example 2.

  input_t = Input((4, 2))
output_t = LSTM(3)(input_t)
model = Model(inputs=input_t, outputs=output_t)
print(model.summary())

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_6 (InputLayer)         (None, 4, 2)              0
_________________________________________________________________
lstm_6 (LSTM)                (None, 3)                 72
=================================================================
Total params: 72
Trainable params: 72
Non-trainable params: 0


Number of units is 3, the size of the input vector is 2, so $4\times 3 \times (2 + 3 +1) = 72$

I usually work with Tensorflow but I as I could see in the documentation it's similar to Keras.

Basically, when you are calling LSTM(3) you are NOT creating LSTM one top of each other like on this image 1. This is a completely different problem.

However, when you are creating LSTM(3) you are making a LSTM with 3 hidden units or hidden cells. On your code, 3 will be the dimension of the inner cells in LSTM. What does it means? This means that the dimensionality of the hidden state and the dimensionality of the output state will be the same as your parameter of hidden units.

Instead of imagine a LSTM as something that get a sequence of scalars and give and output scalar, imagine this: You have a sequence of T length with 512 values each T so [Batchsize, T, 512]. At the first timestemp T=1, you will feed the LSTM with these 512 values at once and this is thanks to the hidden units.

I attach you some references and links if my explanation is not very clear. Q Reference, S Reference.

I made a picture (see sources for original pictures) showing cells as classically represented in tutorials (Source1: Colah's Blog) and an equivalent cell with 2 units (Source2: Raimi Karim 's post). Hope it will clarify confusion between cells/units and what really is the network architecture.