48
$\begingroup$

I have been studying LSTMs for a while. I understand at a high level how everything works. However, going to implement them using Tensorflow I've noticed that BasicLSTMCell requires a number of units (i.e. num_units) parameter.

From this very thorough explanation of LSTMs, I've gathered that a single LSTM unit is one of the following

LSTM unit

which is actually a GRU unit.

I assume that parameter num_units of the BasicLSTMCell is referring to how many of these we want to hook up to each other in a layer.

That leaves the question - what is a "cell" in this context? Is a "cell" equivalent to a layer in a normal feed-forward neural network?

$\endgroup$
2
  • $\begingroup$ I am still confused, I was reading colah.github.io/posts/2015-08-Understanding-LSTMs and I understand that well. How does the term cell apply with respect to that article? It seems that an LSTM cell in the article is a vector as in Tensorflow, right? $\endgroup$ Apr 19 '17 at 4:44
  • $\begingroup$ That units in Keras is the dimension of the output space, which is equal to the length of the delay (time_step) the network is recurring to. keras.layers.LSTM(units, activation='tanh', ....) keras.io/layers/recurrent $\endgroup$
    – notilas
    May 28 '19 at 23:12
25
$\begingroup$

The terminology is unfortunately inconsistent. num_units in TensorFlow is the number of hidden states, i.e. the dimension of $h_t$ in the equations you gave.

Also, from https://github.com/tensorflow/tensorflow/blob/master/tensorflow/g3doc/api_docs/python/functions_and_classes/shard9/tf.nn.rnn_cell.RNNCell.md :

The definition of cell in this package differs from the definition used in the literature. In the literature, cell refers to an object with a single scalar output. The definition in this package refers to a horizontal array of such units.

"LSTM layer" is probably more explicit, example:

def lstm_layer(tparams, state_below, options, prefix='lstm', mask=None):
    nsteps = state_below.shape[0]
    if state_below.ndim == 3:
        n_samples = state_below.shape[1]
    else:
        n_samples = 1

    assert mask is not None
    […]
$\endgroup$
9
  • $\begingroup$ Ah I see, so then a "cell" is a num_unit sized horizontal array of interconnected LSTM cells. Makes sense. So then it would be analogous to a hidden layer in a standard feed-forward network then? $\endgroup$
    – user124589
    Oct 24 '16 at 1:32
  • $\begingroup$ *LSTM state units $\endgroup$
    – user124589
    Oct 24 '16 at 1:33
  • $\begingroup$ @rec That's correct $\endgroup$ Oct 24 '16 at 1:33
  • 1
    $\begingroup$ @Sycorax for example, if the input of the neural network is a timeseries with 10 time steps, the horizontal dimension has 10 elements. $\endgroup$ Oct 24 '16 at 3:36
  • 1
    $\begingroup$ I am still confused, I was reading colah.github.io/posts/2015-08-Understanding-LSTMs and I understand that well. How does the term cell apply with respect to that article? It seems that an LSTM cell in the article is a vector as in Tensorflow, right? $\endgroup$ Apr 19 '17 at 4:44
8
$\begingroup$

Most LSTM/RNN diagrams just show the hidden cells but never the units of those cells. Hence, the confusion. Each hidden layer has hidden cells, as many as the number of time steps. And further, each hidden cell is made up of multiple hidden units, like in the diagram below. Therefore, the dimensionality of a hidden layer matrix in RNN is (number of time steps, number of hidden units).

enter image description here

$\endgroup$
5
$\begingroup$

Although the issue is almost the same as I answered in this answer, I'd like to illustrate this issue, which also confused me a bit today in the seq2seq model (thanks to @Franck Dernoncourt's answer), in the graph. In this simple encoder diagram:

enter image description here

Each $h_i$ above is the same cell in different time-step (cell either GRU or LSTM as that in your question) and the weight vectors(not bias) in the cell are of the same size of (num_units/num_hidden or state_size or output_size).

RNN is a special type of graphical model where nodes form a directed list as explained in section 4 of this paper: Supervised Neural Networks for the Classication of Structures. We can think of num_units as the number of tags in CRF(although CRF is undirected), and the matrices($W$'s in graph in the question) are all shared across all time steps like the transition matrix in CRF.

$\endgroup$
2
  • $\begingroup$ I believe the num_units = n in this figure $\endgroup$
    – notilas
    May 28 '19 at 23:10
  • 1
    $\begingroup$ @notilas No, please don't. num_units is the dimension of the $h_i$. $\endgroup$ Mar 5 '20 at 4:25
2
$\begingroup$

In keras.layers.LSTM(units, activation='tanh', ....), the units refers to the dimensionality or length of the hidden state or the length of the activation vector passed on the next LSTM cell/unit - the next LSTM cell/unit is the "green picture above with the gates etc from http://colah.github.io/posts/2015-08-Understanding-LSTMs/

The next LSTM cell/unit (i.e. the green box with gates etc from http://colah.github.io/posts/2015-08-Understanding-LSTMs/) is NOT the same as the units in keras.layers.LSTM(units, activation='tanh', ....)

The units are also sometimes called the latent dimensions. Here is a detailed explanation of the units LSTM parameter:

https://zhuanlan.zhihu.com/p/58854907

$\endgroup$
1
  • $\begingroup$ Thank you. Your last link has the clearest explanation I've seen thus far. I held many of the same misconceptions that the author clears up. $\endgroup$
    – Kevin D.
    May 15 at 20:23
0
$\begingroup$

Quoting from TF's tutorial on RNNs:

In addition to the built-in RNN layers, the RNN API also provides cell-level APIs. Unlike RNN layers, which processes whole batches of input sequences, the RNN cell only processes a single timestep.

$\endgroup$
-1
$\begingroup$

In my opinion, cell means a node such as hidden cell which is also called hidden node, for multilayer LSTM model,the number of cell can be computed by time_steps*num_layers, and the num_units is equal to time_steps

$\endgroup$