# Batch normalization: How to update gamma and beta during backpropagation training step?

The backpropagation step of batch normalization computes the derivative of gamma (let's call it dgamma) and the derivative of beta (let's call it dbeta) among with dx, the actual gradient for the loss signal.

For those who don't remember, gamma was used to scale the normalized values and beta was used to shift them up or down (which eliminates the need for bias)

The original backpropagation paper says that gamma and beta are learnable parameters but it doesn't say how to learn them. I would assume that the returned dbeta and dgamma are needed to update gamma and beta. But I couldn't find any example showing how that's done. My intuition tells me that I should update them in the same way as I updated the weights, using the same learning rate as used for weights. By which I mean something like

gamma_updated = gamma - learning_rate*dgamma
beta_updated = beta - learning_rate*dbeta


But this isn't specified anywhere, so I don't know if my intuition is correct. For example I could as well use a separate learning rate. I could also apply some function on dbeta and dgamma or I could find an entirely different way to update them.

So how are gamma and beta updated in practice?

• Your intuition is correct. The gamma and beta are updated exactly like other parameters in the network. – Hossein Nov 5 '17 at 20:02
• Thanks! Is this a general rule in practice or rather an exception in case of BN? Can this be assumed for arbitrary vector functions (besides BN) as long as the parameters are differentiable? Or is BN one of the few functions where you can learn the parameters like that. I'm thinking for example about flexible ReLu's where you can learn the gradient of the linear part instead of fixing it at 1. Don't remember if Softmax had learnable parameters as well but I believe not. – evolution Nov 5 '17 at 20:18
• Look at the answer. About the ReLU, it is not necessary to learn its slope, since the weight parameter associated with the connection from its output to the next layer plays this role. – Hossein Nov 6 '17 at 17:54