The backpropagation step of batch normalization computes the derivative of
gamma (let's call it
dgamma) and the derivative of
beta (let's call it
dbeta) among with
dx, the actual gradient for the loss signal.
For those who don't remember,
gamma was used to scale the normalized values and
beta was used to shift them up or down (which eliminates the need for bias)
The original backpropagation paper says that
beta are learnable parameters but it doesn't say how to learn them. I would assume that the returned
dgamma are needed to update
beta. But I couldn't find any example showing how that's done. My intuition tells me that I should update them in the same way as I updated the weights, using the same learning rate as used for weights. By which I mean something like
gamma_updated = gamma - learning_rate*dgamma beta_updated = beta - learning_rate*dbeta
But this isn't specified anywhere, so I don't know if my intuition is correct. For example I could as well use a separate learning rate. I could also apply some function on
dgamma or I could find an entirely different way to update them.
So how are
beta updated in practice?