Why are bias nodes used in neural networks?

Question

1. Why are bias nodes used in neural networks?
2. How many you should use?
3. In which layers you should use them: all hidden layers and the output layer?

Answer 1

The bias node in a neural network is a node that is always 'on'. That is, its value is set to $1$ without regard for the data in a given pattern. It is analogous to the intercept in a regression model, and serves the same function. If a neural network does not have a bias node in a given layer, it will not be able to produce output in the next layer that differs from $0$ (on the linear scale, or the value that corresponds to the transformation of $0$ when passed through the activation function) when the feature values are $0$.

Consider a simple example: You have a feed forward perceptron with 2 input nodes $x_1$ and $x_2$, and 1 output node $y$. $x_1$ and $x_2$ are binary features and set at their reference level, $x_1=x_2=0$. Multiply those 2 $0$'s by whatever weights you like, $w_1$ and $w_2$, sum the products and pass it through whatever activation function you prefer. Without a bias node, only one output value is possible, which may yield a very poor fit. For instance, using a logistic activation function, $y$ must be $.5$, which would be awful for classifying rare events.

A bias node provides considerable flexibility to a neural network model. In the example given above, the only predicted proportion possible without a bias node was $50\%$, but with a bias node, any proportion in $(0, 1)$ can be fit for the patterns where $x_1=x_2=0$. For each layer, $j$, in which a bias node is added, the bias node will add $N_{j+1}$ additional parameters / weights to be estimated (where $N_{j+1}$ is the number of nodes in layer $j+1$). More parameters to be fitted means it will take proportionately longer for the neural network to be trained. It also increases the chance of overfitting, if you don't have considerably more data than weights to be learned.

With this understanding in mind, we can answer your explicit questions:

1. Bias nodes are added to increase the flexibility of the model to fit the data. Specifically, it allows the network to fit the data when all input features are equal to $0$, and very likely decreases the bias of the fitted values elsewhere in the data space.
2. Typically, a single bias node is added for the input layer and every hidden layer in a feedforward network. You would never add two or more to a given layer, but you might add zero. The total number is thus determined largely by the structure of your network, although other considerations could apply. (I am less clear on how bias nodes are added to neural network structures other than feedforward.)
3. Mostly this has been covered, but to be explicit: you would never add a bias node to the output layer; that wouldn't make any sense.

Answer 2

In the context of neural networks, Batch Normalization is currently the gold-standard for making smart "bias nodes." Instead of clamping a neuron's bias value, you instead adjust for the covariance of the neuron's input. So in a CNN, you would apply a batch normalization just between the convolutional layer and the next fully connected layer (of say, ReLus). In theory, all fully connected layers could benefit from Batch Normalization but this in practice becomes very expensive to implement since each batch normalization carries its own parameters.

Concerning why, most of the answers already have explained that, in particular, neurons are susceptible to saturated gradients when the input pushes the activation to an extreme. In the case of ReLu's this would be pushed to the left, giving a gradient of 0. In general, when you train a model, you first normalize the inputs to the neural network. Batch Normalization is a way of normalizing the inputs inside the neural network, between layers.

Answer 3

Simple, short answers:

1. To shift the input function / be more flexible about the learned function.
2. A single bias node per layer.
3. Add them to all hidden layers and the input layer - with some footnotes

In a couple of experiments in my masters thesis (e.g. page 59), I found that the bias might be important for the first layer(s), but especially at the fully connected layers at the end it seems not to play a big role. Hence one can have them at the first few layers and not at the last ones. Simply train a network, plot the distribution of weights of the bias nodes and prune them if the weights seem to be too close to zero.

This might be highly dependent on the network architecture / dataset.