Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
What is the correct way to initialize biases in convolutional neural networks (tf.zeros, tf.truncated_normal, tf.random_normal), and why?
Should biases be constant? Do we even need biases in a deep neural network (like VGG-16)?
In a siamese neural network, do we also share the biases along with the weights?
Just noting that the answer to this question suggests setting CNN biases to 0, quoting CS231n Stanford course:
Initializing the biases. It is possible and common to initialize the biases to be zero, since the asymmetry breaking is provided by the small random numbers in the weights. For ReLU non-linearities, some people like to use small constant value such as 0.01 for all biases because this ensures that all ReLU units fire in the beginning and therefore obtain and propagate some gradient. However, it is not clear if this provides a consistent improvement (in fact some results seem to indicate that this performs worse) and it is more common to simply use 0 bias initialization.
Usually you initialise them to 1.0
Biases should be trainable variables not constant, their value must be allowed to change during training. Biases are necessary in every deep network architecture I know of, without them your network will most likely be unable to learn anything.
I don't know what a siamese neural network is but in architectures where weights are shared (such convolution neural networks) weights and biases are always shared together as they come in pairs, the combination of the two is what defines a layer.
