Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
When building an artificial neural network, there seems to be two differing philosophies in usage of biases. There are those groups that propose neural networks with a fixed bias neuron with a variable weight, and those who propose variable bias values for each neuron in the hidden and output layers. Which one is better? Is one really better? (Advantages, disadvantages...) Also see the examples below.
In principle, you're just expressing the same mathematical model with a slightly different form. In practice, the difference lurks in two aspects, initialization and learning.
Biases are typically initialized at 0 for symmetric activation function, or at some positive value for rectified linear units. Using a fixed bias neuron means that you will be using the weight's initialization procedure to pick the biases, which will typically yield a different result.
It is also common to use a larger learning rates for biases than for weights. Though there doesn't seem to be much theoretical justification around it, this seems to be a useful heuristic born out of experimentation by practitioners.
I'll give an answer in two parts:
How can one assign bias to nodes? Both of the situations you depict are identical. Although in the first drawing the bias is always one, the arrows to the nodes will be assigned a coefficient in the model: resulting in different biases for each node. Hence, there is no difference in this sense between your first and second drawing.
Should bias be added to both hidden and output layers? Normally bias is added to all layers, but this is not a principled choice: it just results in the most flexible model. Removing bias from the hidden layers might work out. In this case you would have to show that this model with less parameters performs as well or better as the model with the extra bias in the hidden layer.
In general a sparser model is seen as a better model. In the case of neural networks though is not that common to remove parameters for reasons of sparsity. Shrinkage is more common way to make your nn model sparser.
