# Batch normalization and the need for bias in neural networks

I've read that batch normalization eliminates the need for a bias vector in neural networks, since it introduces a shift parameter that functions similarly as a bias. As far as I'm aware though, a bias term works on a per-node level whereas the shift parameter of batch normalization is applied to all the activations. For instance in convolutional neural networks, the bias vector can be seen as an extra receptive field input of which the input is always 1. This effectively shifts each individual activation as opposed to shifting all activations at once.

My question thus is: is it true that batch normalization eliminates the need for a bias vector?

• Someone can correct me if I'm wrong, but bias term is a very cheap parameter to have, so you probably wouldn't care that much about it.
– Tim
Commented May 12, 2021 at 10:12
• Batch norm has vectors for both the shift and scale parameters. stats.stackexchange.com/questions/414630/… Once this misunderstanding is corrected, OP's explanation of the redundancy of multiple bias vectors is correct and the logic is sound.
– Sycorax
Commented Aug 15, 2023 at 15:11

Check your software, but broadly the answer to your question is: yes, using batch normalization obviates the need for a bias in the preceding linear layer. Your question does a good job of laying out where you are confused, so let me speak to it: the shift term in batch normalization is also a vector, for instance the documentation for BatchNorm2d in Pytorch reads: "The mean and standard-deviation are calculated per-dimension over the mini-batches and $$\gamma$$ and $$\beta$$ are learnable parameter vectors of size $$C$$ (where $$C$$ is the input size). "
Yes. Let $$x'=wx+b$$. Then $$E[x']=wE[x]+b$$ and $$x'-E[x']$$ becomes $$wx+b-wE[x]-b$$ and $$b$$ gets canceled. And $$\beta$$ does the job of $$b$$.