2
$\begingroup$

I have been reading about RNNs and I have some confusion between the number of timesteps and number of units in an RNN layer (which after searching for an answer) seems to be a common thing.
I understand that the idea behind using a RNN is to capture the relationship between every element in the sequence with other elements. In an example of text classification with tweets padded to a length of 20 words, the number of units in a layer -which I understand is the same as the number of timesteps- should be equal to the length of the input sequence which is 20.
I also saw some examples that deals with the number of units as a hyperparameter to be tuned.
Which of these is correct?

enter image description here

This picture is from a course on coursera by Andrew Ng. The super script Tx indicates the number of timesteps which is the same as the number of units in the network.

$\endgroup$

1 Answer 1

4
$\begingroup$

The number of cells and sequence length are distinct concepts. I think the clearest way to demonstrate this is to just write out the equations. For instance, the simple Elman RNN has equations

$$ \begin{align} a_t &= f_a\left(W_a x_t + U_a a_{t-1} + b_a\right) \\ y_t &= f_y\left(W_y a_t + b_y\right) \end{align} $$

where $a_t$ is the hidden state vector at time $t$, $y_t$ is the prediction at time $t$ and $x_t$ is the input at time $t$. Each $x_t$ is a vector with some number of features $k$, and the hidden state has some dimension $p$. So $W_a$ must have shape $p \times k$ for the matrix multiplication to work. Likewise, matrix $U_a$ must have shape $p\times p$ and the bias $b_a$ must have shape $p\times 1$. The functions $f_a, f_y$ are just the activation function(s) that you're using -- sigmoid or ReLU or any other activation.

If you wish to apply an RNN to a sequence, first initialize $a_0$ and then loop over the recurrence relation. In psuedocode, it looks something like this:

for t in 1 ... T:
    a[t] = f_a(x[t], a[t-1])
    y[t] = f_y(a[t])

where f_a and f_y apply the weights and biases as well as the non-linearity.

The number of units is $p$. It should be clear that you can change $p$ and the total number of time steps $T$ independently, because the recurrence relation allows us to carry out the prediction loop for as many steps as we wish -- as long as we can provide $x_t$.

We can do the same for the and networks.

$\endgroup$
10
  • $\begingroup$ Why would we need more units than the length of the sequence? And in such case, how is each sequence passed through the layer? Assume a sequence with 20 words and a layer with 10 units, the sequence is passed through the layer in two pieces with 10 words each. Is this correct? $\endgroup$ Commented Apr 29, 2021 at 17:42
  • $\begingroup$ And assume another case where the sequence is of 20 words and the layer has 30 units, is the sequence passed through the first 20 units then the next sequence comes in? $\endgroup$ Commented Apr 29, 2021 at 17:44
  • $\begingroup$ The descriptions in these comments are not correct because time and number of units are distinct concepts, as described in my answer. The dimension of the weight matrices does not reflect time. The concept of time is defined by the recurrence relation. $\endgroup$
    – Sycorax
    Commented Apr 29, 2021 at 17:48
  • $\begingroup$ what do you mean by recurrence relation? $\endgroup$ Commented Apr 29, 2021 at 17:51
  • $\begingroup$ The value of $a_t$ depends on $a_{t-1}$ (and $x_t$, but dependence on $x_t$ is not a recurrence). $\endgroup$
    – Sycorax
    Commented Apr 29, 2021 at 17:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.