Bias inputs in an RNN

Question

As far as I'm aware, the bias inputs for a feed forward neural network are typically connected as follows:

How are they connected in a recurrent neural network? (My guess is below)

Answer 1

This is basically correct.

The bias is an "offset" added to each unit in a neural network layer that's independent of the input to the layer. The bias permits a layer to model a data space that's centered around some point other than the origin.

Mathematically, a feedforward neural network layer without bias is written as $$ z = \sigma(Wx) $$ where $W$ is the weights of the layer, $x$ is the input to a layer, and $\sigma(\cdot)$ is the activation function for the layer. If you want to add a bias to this expression, it's common to create a separate parameter $$ z = \sigma(Wx + \color{red}{b}) = \sigma\left(\sum_{i=1}^n W_i x_i + \color{red}{b}\right). $$ But this is equivalent to creating a pseudo-input node $\color{red}{x_0}=1$ in the previous layer and stacking it onto the input so $\hat{x} = [\color{red}{1} \; x^\top]^\top$, with $\color{red}{b}$ being stacked onto the start of $W$ so $\hat{W} = [\color{red}{b} \; W]$: $$ z = \sigma(\hat{W}\hat{x}) = \sigma\left(\sum_{i=0}^n \hat{W}_i \hat{x}_i\right) = \sigma\left(\color{red}{\hat{W}_0 \cdot 1} + \sum_{i=1}^n \hat{W}_i \hat{x}_i\right) $$

A recurrent network layer is basically the same. Without a bias, the output $z_t$ is given by $$ z_t = \sigma(Wx_t + Vz_{t-1}) $$ where $V$ is an array of weights that connects the previous state of the hidden layer to the current state. Adding a bias gives the recurrent layer this form: $$ z_t = \sigma(Wx_t + Vz_{t-1} + \color{red}{b}). $$