7
$\begingroup$

What is the correct way to initialize biases in convolutional neural networks (tf.zeros, tf.truncated_normal, tf.random_normal), and why?

Should biases be constant? Do we even need biases in a deep neural network (like VGG-16)?

In a siamese neural network, do we also share the biases along with the weights?

$\endgroup$
1
  • $\begingroup$ I thought however Xavier init did it was the standard way...I'm not sure if initialization of W or b matters that much anymore with the existence of batch-normalization... $\endgroup$ – Charlie Parker Jan 11 '18 at 15:02
4
$\begingroup$

Just noting that the answer to this question suggests setting CNN biases to 0, quoting CS231n Stanford course:

Initializing the biases. It is possible and common to initialize the biases to be zero, since the asymmetry breaking is provided by the small random numbers in the weights. For ReLU non-linearities, some people like to use small constant value such as 0.01 for all biases because this ensures that all ReLU units fire in the beginning and therefore obtain and propagate some gradient. However, it is not clear if this provides a consistent improvement (in fact some results seem to indicate that this performs worse) and it is more common to simply use 0 bias initialization.

source: http://cs231n.github.io/neural-networks-2/

$\endgroup$
1
  • $\begingroup$ I knew I saw the b=0.01 somewhere! Good job for digging a "reputable" source that mentions it and its rationale. $\endgroup$ – Charlie Parker Jan 11 '18 at 15:05
0
$\begingroup$

Usually you initialise them to 1.0

Biases should be trainable variables not constant, their value must be allowed to change during training. Biases are necessary in every deep network architecture I know of, without them your network will most likely be unable to learn anything.

I don't know what a siamese neural network is but in architectures where weights are shared (such convolution neural networks) weights and biases are always shared together as they come in pairs, the combination of the two is what defines a layer.

$\endgroup$
6
  • $\begingroup$ Can you provide a reference or a reason for your statement that they are initialized with a value of 1 ? $\endgroup$ – meh Sep 21 '17 at 12:33
  • $\begingroup$ I don't have a reference, but the intuition behind it being 1 and not 0 is that if it were 0 it would be as if they did not exist. This makes learning harder due to the fact that as I said biases are strictly necessary for learning. If they start at 0 and are trainable your learning algorithm should in principle be able to change the values to something other than 0, I just think it would be slower. $\endgroup$ – Miguel Sep 21 '17 at 12:38
  • $\begingroup$ For this I am assuming that your data is already scaled to zero mean and unit variance. Depending on the particular architecture there can be better ways choosing the initialisation of weights and biases, however the scientific discussion mostly focuses on the weights. $\endgroup$ – Miguel Sep 21 '17 at 12:41
  • $\begingroup$ @Miguel Can I initialize the biases using xavier? Does it make sense? $\endgroup$ – Hello Lili Sep 21 '17 at 13:30
  • $\begingroup$ I don't see why not, should be fine. I usually use xavier just for the weights. Maybe it will help converge faster maybe it won't do much, if you do end up testing both approaches I would be interested in hearing the results. $\endgroup$ – Miguel Sep 21 '17 at 13:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.