Bias initialization in convolutional neural network

What is the correct way to initialize biases in convolutional neural networks (tf.zeros, tf.truncated_normal, tf.random_normal), and why?

Should biases be constant? Do we even need biases in a deep neural network (like VGG-16)?

In a siamese neural network, do we also share the biases along with the weights?

• I thought however Xavier init did it was the standard way...I'm not sure if initialization of W or b matters that much anymore with the existence of batch-normalization... Jan 11, 2018 at 15:02

Just noting that the answer to this question suggests setting CNN biases to 0, quoting CS231n Stanford course:

Initializing the biases. It is possible and common to initialize the biases to be zero, since the asymmetry breaking is provided by the small random numbers in the weights. For ReLU non-linearities, some people like to use small constant value such as 0.01 for all biases because this ensures that all ReLU units fire in the beginning and therefore obtain and propagate some gradient. However, it is not clear if this provides a consistent improvement (in fact some results seem to indicate that this performs worse) and it is more common to simply use 0 bias initialization.

• I knew I saw the b=0.01 somewhere! Good job for digging a "reputable" source that mentions it and its rationale. Jan 11, 2018 at 15:05

Usually you initialise them to 1.0

Biases should be trainable variables not constant, their value must be allowed to change during training. Biases are necessary in every deep network architecture I know of, without them your network will most likely be unable to learn anything.

I don't know what a siamese neural network is but in architectures where weights are shared (such convolution neural networks) weights and biases are always shared together as they come in pairs, the combination of the two is what defines a layer.

• Can you provide a reference or a reason for your statement that they are initialized with a value of 1 ?
– meh
Sep 21, 2017 at 12:33
• I don't have a reference, but the intuition behind it being 1 and not 0 is that if it were 0 it would be as if they did not exist. This makes learning harder due to the fact that as I said biases are strictly necessary for learning. If they start at 0 and are trainable your learning algorithm should in principle be able to change the values to something other than 0, I just think it would be slower. Sep 21, 2017 at 12:38
• For this I am assuming that your data is already scaled to zero mean and unit variance. Depending on the particular architecture there can be better ways choosing the initialisation of weights and biases, however the scientific discussion mostly focuses on the weights. Sep 21, 2017 at 12:41
• @Miguel Can I initialize the biases using xavier? Does it make sense? Sep 21, 2017 at 13:30
• I don't see why not, should be fine. I usually use xavier just for the weights. Maybe it will help converge faster maybe it won't do much, if you do end up testing both approaches I would be interested in hearing the results. Sep 21, 2017 at 13:33