According to this tutorial on deep learning, weight decay (regularization) is not usually applied to the bias terms b why?
What is significance (intuition) behind it?
$\begingroup$
$\endgroup$
3
-
$\begingroup$ I think I have seen a very similar question before, I just cannot find it... Perhaps you should review related questions and would find the answer then. Also, perhaps this could be somewhat useful. $\endgroup$– Richard HardyMay 27, 2015 at 19:35
-
$\begingroup$ I don't agree with the responders. One thing weight decay is for is normalization. But when you change just the weights and not the bias you also fundamentally change your layer's output, especially given that activation follows, not just scale it. I'll have to experiment, but I think it's proper to scale down the bias as well. $\endgroup$– Íhor MéAug 5, 2020 at 20:00
-
$\begingroup$ The link is an empty wiki page at the moment... $\endgroup$– Maverick MeerkatJun 19, 2022 at 13:53
Add a comment
|
5 Answers
$\begingroup$
$\endgroup$
Overfitting usually requires the output of the model to be sensitive to small changes in the input data (i.e. to exactly interpolate the target values, you tend to need a lot of curvature in the fitted function). The bias parameters don't contribute to the curvature of the model, so there is usually little point in regularising them as well.
answered Aug 18, 2015 at 8:08
$\begingroup$
$\endgroup$
The motivation behind L2 (or L1) is that by restricting the weights, constraining the network, you are less likely to overfit. It makes little sense to restrict the weights of the biases since the biases are fixed (e.g. b = 1) thus work like neuron intercepts, which make sense to be given a higher flexibility.
$\begingroup$
$\endgroup$
Weights determine slopes of the activation functions. Regularization reduces the weights and hence the slopes of the activation functions. This reduces the model variance and the overfitting effect. The biases have no influence on the slopes of activation functions. However, they have an influence on the position of the activation functions in space. Their optimal values depend on the weights, so they should be adjusted to the regularized weights. The biases should be adjusted without regularization. Their regularization can be harmful. I considered the functions of weights and biases in randomized NN, see here
$\begingroup$
$\endgroup$
2
I would add that the bias term is often initialized with a mean of 1 rather than of 0, so we might want to regularize it in a way to not get too far away from a constant value like 1 such as doing 1/2*(bias-1)^2 rather than 1/2*(bias)^2.
Maybe that replacing the -1 part by a subtraction to the mean of the biases could help, maybe a per-layer mean or an overall one. Yet this is just a hypothesis I am doing (about the mean substraction).
This all depends on the activation function too. E.g.: sigmoids might be bad here for vanishing gradients if biases are regularized to high constant offsets.
-
1$\begingroup$ Who initialize their biases with 1? That seems like a sure way to kill most of your relus out of the gate. $\endgroup$ May 10 at 13:36
-
1$\begingroup$ Killing relus is more likely to happen with a negative bias than with a positive bias. $\endgroup$ Jun 13 at 3:51
$\begingroup$
$\endgroup$
The tutorial says "applying weight decay to the bias units usually makes only a small difference to the final network", so if it does not help, then you can stop doing it to eliminate one hyperparameter. If you think regularizing the offset would help in your setup, then cross-validate it; there's no harm in trying.
