# No regularisation term for bias unit in neural network

According to this tutorial on deep learning, weight decay (regularization) is not usually applied to the bias terms b why?

What is significance (intuition) behind it?

• I think I have seen a very similar question before, I just cannot find it... Perhaps you should review related questions and would find the answer then. Also, perhaps this could be somewhat useful. – Richard Hardy May 27 '15 at 19:35
• I don't agree with the responders. One thing weight decay is for is normalization. But when you change just the weights and not the bias you also fundamentally change your layer's output, especially given that activation follows, not just scale it. I'll have to experiment, but I think it's proper to scale down the bias as well. – Íhor Mé Aug 5 at 20:00

I would add that the bias term is often initialized with a mean of 1 rather than of 0, so we might want to regularize it in a way to not get too far away from a constant value like 1 such as doing 1/2*(bias-1)^2 rather than 1/2*(bias)^2.
Maybe that replacing the -1 part by a subtraction to the mean of the biases could help, maybe a per-layer mean or an overall one. Yet this is just a hypothesis I am doing (about the mean substraction).