Understanding Recurrent Neural Networks

Question

I am trying to understand RNNs and then trying to implement them using batch SGD:

I am stuck in understanding how it will work:

RNN has $t$ time steps, $(X_i)$ is the input at each time step, $h$ is the hidden state, then RNN is just a linear layer with 2 inputs x and a hidden input (i.e 2 matrices).

And if we do:

for x in sequence_length:
    h = linear(x, h)

This actually means recurrence.

1. What will be the output at the end of this RNN?

2. I want to understand, how to do batching here? When doing batching what will the input look like?

   [batch, seq, embedding_size] or [seq, batch, embedding_size]

3. And how to extend to multiple layers of RNN? What do multiple layers even mean in case of RNNs... aren't RNN layers just determined by the sequence_length? So, how to implement multiple layer RNNs.

All software packages just don't answer these questions.

All tutorials just start from advanced without explaining these two questions.

Answer 1

What will be the output at the end of this RNN?

This might be different, because there're different flavors of RNN (most popular are LSTM and GRU), but in the simplest case the current hidden vector is the output of this cell:

$$o_t = h_t = F(x_t, h_{t-1})$$

Note that there are many cells in a row, so the output shape is [seq_length, hidden_size]. Depending on your task, you may want to ignore the output of all cells except for the last one, because they haven't seen the whole input. In this case, the output would be just [hidden_size].

I want to understand, how to do batching here? When doing batching what will the input look like?

Just like in the ordinary neural network. You can feed the sequences ${x_t^{(1)}}$ and ${x_t^{(2)}}$ one by one or in a single batch, computing both simultaneously.

Then the input shape would be [batch_size, seq_length, input_size] and the output shape would be [batch_size, seq_length, hidden_size]. Some implementations may force you to swap seq_length and batch_size due to internal details, but I'm not aware of any such implementation. Tensorflow and Keras have the batch_size the first dimension.

And how to extend to multiple layers of RNN? What do multiple layers even mean in case of RNNs... aren't RNN layers just determined by the sequence_length? So, how to implement multiple layer RNNs.

Deep RNN means that stacking two or more sequences of cells. The output of the first RNN goes as input to the next one, and so on.

Each layer is just like ordinary RNN: a cell takes an input sequence and the previous hidden state. The difference that deeper layers receive not the raw input $x_t$, but a transformed representation of it after it's gone through a previous layer. In this sense, it's not different from feed-forward neural networks with two or more layers.