# Bias inputs in an RNN

As far as I'm aware, the bias inputs for a feed forward neural network are typically connected as follows:

How are they connected in a recurrent neural network? (My guess is below)

• Feed forward Neural Network and Recurrent Neural Network - I've just edited the question – dok Aug 29 '15 at 19:12

Mathematically, a feedforward neural network layer without bias is written as $$z = \sigma(Wx)$$ where $W$ is the weights of the layer, $x$ is the input to a layer, and $\sigma(\cdot)$ is the activation function for the layer. If you want to add a bias to this expression, it's common to create a separate parameter $$z = \sigma(Wx + \color{red}{b}) = \sigma\left(\sum_{i=1}^n W_i x_i + \color{red}{b}\right).$$ But this is equivalent to creating a pseudo-input node $\color{red}{x_0}=1$ in the previous layer and stacking it onto the input so $\hat{x} = [\color{red}{1} \; x^\top]^\top$, with $\color{red}{b}$ being stacked onto the start of $W$ so $\hat{W} = [\color{red}{b} \; W]$: $$z = \sigma(\hat{W}\hat{x}) = \sigma\left(\sum_{i=0}^n \hat{W}_i \hat{x}_i\right) = \sigma\left(\color{red}{\hat{W}_0 \cdot 1} + \sum_{i=1}^n \hat{W}_i \hat{x}_i\right)$$
A recurrent network layer is basically the same. Without a bias, the output $z_t$ is given by $$z_t = \sigma(Wx_t + Vz_{t-1})$$ where $V$ is an array of weights that connects the previous state of the hidden layer to the current state. Adding a bias gives the recurrent layer this form: $$z_t = \sigma(Wx_t + Vz_{t-1} + \color{red}{b}).$$