When building an artificial neural network, there seems to be two differing philosophies in usage of biases. There are those groups that propose neural networks with a fixed bias neuron with a variable weight, and those who propose variable bias values for each neuron in the hidden and output layers. Which one is better? Is one really better? (Advantages, disadvantages...) Also see the examples below.

one bias neuron with fixed value and variable weightsvariable bias neurons

  • $\begingroup$ aren't they mathematically identical? $\endgroup$
    – Memming
    Jul 13 '15 at 17:23
  • $\begingroup$ well, now that you're saying it... the activation of k1 is the inputs from layer j to k1 times weights plus bias=1 * weight (case 1) or plus bias (case 2), so that's mathematically the same. but do they learn the same way? I'm not sure about that $\endgroup$
    – cirko
    Jul 14 '15 at 10:24

In principle, you're just expressing the same mathematical model with a slightly different form. In practice, the difference lurks in two aspects, initialization and learning.

Biases are typically initialized at 0 for symmetric activation function, or at some positive value for rectified linear units. Using a fixed bias neuron means that you will be using the weight's initialization procedure to pick the biases, which will typically yield a different result.

It is also common to use a larger learning rates for biases than for weights. Though there doesn't seem to be much theoretical justification around it, this seems to be a useful heuristic born out of experimentation by practitioners.

  • $\begingroup$ So summed up, the answer is "There are differences in handling the NN and its parameters, but no real differences in the result"? Plus a little trial and error? :) $\endgroup$
    – cirko
    Jul 7 '15 at 14:49
  • $\begingroup$ There will be differences in the result if you apply the "weight" heuristics to biases, which you will be very tempted to do if you use fixed bias neurons. $\endgroup$
    – Arthur B.
    Jul 7 '15 at 15:01
  • 1
    $\begingroup$ ...so I add "Plus higher η values for bias neurons" to the previous comment, like, 2η or so? $\endgroup$
    – cirko
    Jul 7 '15 at 17:30

I'll give an answer in two parts:

  • how can one assign bias to nodes
  • Should bias be added to both hidden and output layers

How can one assign bias to nodes? Both of the situations you depict are identical. Although in the first drawing the bias is always one, the arrows to the nodes will be assigned a coefficient in the model: resulting in different biases for each node. Hence, there is no difference in this sense between your first and second drawing.

Should bias be added to both hidden and output layers? Normally bias is added to all layers, but this is not a principled choice: it just results in the most flexible model. Removing bias from the hidden layers might work out. In this case you would have to show that this model with less parameters performs as well or better as the model with the extra bias in the hidden layer.

In general a sparser model is seen as a better model. In the case of neural networks though is not that common to remove parameters for reasons of sparsity. Shrinkage is more common way to make your nn model sparser.

  • $\begingroup$ I didn't intend to leave out the bias for the hidden layer in the first picture, it just occurred to me after you mentioned it $\endgroup$
    – cirko
    Jul 7 '15 at 14:47
  • $\begingroup$ "Normally bias is added to all layers". That is incorrect. A bias at the first layer, the input layer, is redundant and is therefor usually not included - because why would you? $\endgroup$ Oct 22 '18 at 17:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.