Number of cells in an RNN

Question

I have been reading about RNNs and I have some confusion between the number of timesteps and number of units in an RNN layer (which after searching for an answer) seems to be a common thing.
I understand that the idea behind using a RNN is to capture the relationship between every element in the sequence with other elements. In an example of text classification with tweets padded to a length of 20 words, the number of units in a layer -which I understand is the same as the number of timesteps- should be equal to the length of the input sequence which is 20.
I also saw some examples that deals with the number of units as a hyperparameter to be tuned.
Which of these is correct?

This picture is from a course on coursera by Andrew Ng. The super script Tx indicates the number of timesteps which is the same as the number of units in the network.

Answer 1

The number of cells and sequence length are distinct concepts. I think the clearest way to demonstrate this is to just write out the equations. For instance, the simple Elman RNN has equations

$$ \begin{align} a_t &= f_a\left(W_a x_t + U_a a_{t-1} + b_a\right) \\ y_t &= f_y\left(W_y a_t + b_y\right) \end{align} $$

where $a_t$ is the hidden state vector at time $t$, $y_t$ is the prediction at time $t$ and $x_t$ is the input at time $t$. Each $x_t$ is a vector with some number of features $k$, and the hidden state has some dimension $p$. So $W_a$ must have shape $p \times k$ for the matrix multiplication to work. Likewise, matrix $U_a$ must have shape $p\times p$ and the bias $b_a$ must have shape $p\times 1$. The functions $f_a, f_y$ are just the activation function(s) that you're using -- sigmoid or ReLU or any other activation.

If you wish to apply an RNN to a sequence, first initialize $a_0$ and then loop over the recurrence relation. In psuedocode, it looks something like this:

for t in 1 ... T:
    a[t] = f_a(x[t], a[t-1])
    y[t] = f_y(a[t])

where f_a and f_y apply the weights and biases as well as the non-linearity.

The number of units is $p$. It should be clear that you can change $p$ and the total number of time steps $T$ independently, because the recurrence relation allows us to carry out the prediction loop for as many steps as we wish -- as long as we can provide $x_t$.

We can do the same for the lstm and gru networks.

• I've answered the same question for LSTM networks. When computing parameters, why is dimensions of hidden-output state of an LSTM-cell assumed same as the number of LSTM-cell?

• I've also reproduced the GRU equations here How many parameters are in a gated recurrent unit (GRU) recurrent neural network (RNN) layer? for a different question.