20
$\begingroup$

According to this tutorial on deep learning, weight decay (regularization) is not usually applied to the bias terms b why?

What is significance (intuition) behind it?

$\endgroup$
  • $\begingroup$ I think I have seen a very similar question before, I just cannot find it... Perhaps you should review related questions and would find the answer then. Also, perhaps this could be somewhat useful. $\endgroup$ – Richard Hardy May 27 '15 at 19:35
  • $\begingroup$ I don't agree with the responders. One thing weight decay is for is normalization. But when you change just the weights and not the bias you also fundamentally change your layer's output, especially given that activation follows, not just scale it. I'll have to experiment, but I think it's proper to scale down the bias as well. $\endgroup$ – Íhor Mé Aug 5 at 20:00
22
$\begingroup$

Overfitting usually requires the output of the model to be sensitive to small changes in the input data (i.e. to exactly interpolate the target values, you tend to need a lot of curvature in the fitted function). The bias parameters don't contribute to the curvature of the model, so there is usually little point in regularising them as well.

| cite | improve this answer | |
$\endgroup$
5
$\begingroup$

The motivation behind L2 (or L1) is that by restricting the weights, constraining the network, you are less likely to overfit. It makes little sense to restrict the weights of the biases since the biases are fixed (e.g. b = 1) thus work like neuron intercepts, which make sense to be given a higher flexibility.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

I would add that the bias term is often initialized with a mean of 1 rather than of 0, so we might want to regularize it in a way to not get too far away from a constant value like 1 such as doing 1/2*(bias-1)^2 rather than 1/2*(bias)^2.

Maybe that replacing the -1 part by a subtraction to the mean of the biases could help, maybe a per-layer mean or an overall one. Yet this is just a hypothesis I am doing (about the mean substraction).

This all depends on the activation function too. E.g.: sigmoids might be bad here for vanishing gradients if biases are regularized to high constant offsets.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

The tutorial says "applying weight decay to the bias units usually makes only a small difference to the final network", so if it does not help, then you can stop doing it to eliminate one hyperparameter. If you think regularizing the offset would help in your setup, then cross-validate it; there's no harm in trying.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.