According to this tutorial on deep learning, weight decay (regularization) is not usually applied to the bias terms b why?
What is significance (intuition) behind it?
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.Sign up to join this community
Overfitting usually requires the output of the model to be sensitive to small changes in the input data (i.e. to exactly interpolate the target values, you tend to need a lot of curvature in the fitted function). The bias parameters don't contribute to the curvature of the model, so there is usually little point in regularising them as well.
The motivation behind L2 (or L1) is that by restricting the weights, constraining the network, you are less likely to overfit. It makes little sense to restrict the weights of the biases since the biases are fixed (e.g. b = 1) thus work like neuron intercepts, which make sense to be given a higher flexibility.
Weights determine slopes of the activation functions. Regularization reduces the weights and hence the slopes of the activation functions. This reduces the model variance and the overfitting effect. The biases have no influence on the slopes of activation functions. However, they have an influence on the position of the activation functions in space. Their optimal values depend on the weights, so they should be adjusted to the regularized weights. The biases should be adjusted without regularization. Their regularization can be harmful. I considered the functions of weights and biases in randomized NN, see here
I would add that the bias term is often initialized with a mean of
1 rather than of
0, so we might want to regularize it in a way to not get too far away from a constant value like
1 such as doing
1/2*(bias-1)^2 rather than
Maybe that replacing the
-1 part by a subtraction to the mean of the biases could help, maybe a per-layer mean or an overall one. Yet this is just a hypothesis I am doing (about the mean substraction).
This all depends on the activation function too. E.g.: sigmoids might be bad here for vanishing gradients if biases are regularized to high constant offsets.
The tutorial says "applying weight decay to the bias units usually makes only a small difference to the final network", so if it does not help, then you can stop doing it to eliminate one hyperparameter. If you think regularizing the offset would help in your setup, then cross-validate it; there's no harm in trying.