4
$\begingroup$

As far as I'm aware, the bias inputs for a feed forward neural network are typically connected as follows:

FNN

How are they connected in a recurrent neural network? (My guess is below)

RNN

$\endgroup$
  • $\begingroup$ Feed forward Neural Network and Recurrent Neural Network - I've just edited the question $\endgroup$ – dok Aug 29 '15 at 19:12
7
$\begingroup$

This is basically correct.

The bias is an "offset" added to each unit in a neural network layer that's independent of the input to the layer. The bias permits a layer to model a data space that's centered around some point other than the origin.

Mathematically, a feedforward neural network layer without bias is written as $$ z = \sigma(Wx) $$ where $W$ is the weights of the layer, $x$ is the input to a layer, and $\sigma(\cdot)$ is the activation function for the layer. If you want to add a bias to this expression, it's common to create a separate parameter $$ z = \sigma(Wx + \color{red}{b}) = \sigma\left(\sum_{i=1}^n W_i x_i + \color{red}{b}\right). $$ But this is equivalent to creating a pseudo-input node $\color{red}{x_0}=1$ in the previous layer and stacking it onto the input so $\hat{x} = [\color{red}{1} \; x^\top]^\top$, with $\color{red}{b}$ being stacked onto the start of $W$ so $\hat{W} = [\color{red}{b} \; W]$: $$ z = \sigma(\hat{W}\hat{x}) = \sigma\left(\sum_{i=0}^n \hat{W}_i \hat{x}_i\right) = \sigma\left(\color{red}{\hat{W}_0 \cdot 1} + \sum_{i=1}^n \hat{W}_i \hat{x}_i\right) $$

A recurrent network layer is basically the same. Without a bias, the output $z_t$ is given by $$ z_t = \sigma(Wx_t + Vz_{t-1}) $$ where $V$ is an array of weights that connects the previous state of the hidden layer to the current state. Adding a bias gives the recurrent layer this form: $$ z_t = \sigma(Wx_t + Vz_{t-1} + \color{red}{b}). $$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.