Understanding number of rnn units in RNN networks

Question

I am trying to learn about recurrent neural networks from here.

There are rnn_units = 1024 in the model and each batch contains 64 sequences.

What I am trying to understand is, do each sentence goes through all 1024 gru units, then an output is produced (GRU ouput)?

In the figure, I feel each character goes to each GRU unit, so I cannot understand how it is working because I think we do not have 1024 characters in each batch. I am a bit confused how this network works.

I appreciate a little more detailed explanation.

Thanks in advance

Answer 1

To answer your question: the sentence does not go through all 1024 GRU units. A better way to put it would be something like every character is converted to a representation with 1024 elements.

The details (in the linked example): Every sentence is to be understood as a sequence of characters. The key is that an RNN actually operates on single characters. If you are familiar with programming, you can think of it as a for loop. Concretely, an RNN generally operates as follows:

0. initialise the state of a pre-specified size (1024)
1. take the next element (char) in the sequence (sentence)
2. map the input element (char) to a representation with the same size as its state (1024)
3. apply a linear transformation on the state (from the previous iteration)
4. compute a (non-linear) transformation on the results from 2 and 3 and make this the new state.
5. jump to step 1 if more elements (chars) are available in the sequence (sentence).

Since, normally, each of the entries in the RNN state vector are independent from each other, they are often considered as individual entities. Therefore, the entries in an RNN state vector are also commonly referred to as units or cells. So, if your RNN has 1024 units, it actually just has a state vector $\in \mathbb{R}^{1024}$.

NOTE: for the sake of simplicity I ignored the fact that every character is normally first transformed/embedded in some vector space (done by tf.keras.layers.Embedding in the example) and these embeddings actually make up the elements of the sequence (instead of the chars).