4
$\begingroup$

The backpropagation step of batch normalization computes the derivative of gamma (let's call it dgamma) and the derivative of beta (let's call it dbeta) among with dx, the actual gradient for the loss signal.

For those who don't remember, gamma was used to scale the normalized values and beta was used to shift them up or down (which eliminates the need for bias)

The original backpropagation paper says that gamma and beta are learnable parameters but it doesn't say how to learn them. I would assume that the returned dbeta and dgamma are needed to update gamma and beta. But I couldn't find any example showing how that's done. My intuition tells me that I should update them in the same way as I updated the weights, using the same learning rate as used for weights. By which I mean something like

gamma_updated = gamma - learning_rate*dgamma
beta_updated = beta - learning_rate*dbeta

But this isn't specified anywhere, so I don't know if my intuition is correct. For example I could as well use a separate learning rate. I could also apply some function on dbeta and dgamma or I could find an entirely different way to update them.

So how are gamma and beta updated in practice?

$\endgroup$
  • $\begingroup$ Your intuition is correct. The gamma and beta are updated exactly like other parameters in the network. $\endgroup$ – Hossein Nov 5 '17 at 20:02
  • $\begingroup$ Thanks! Is this a general rule in practice or rather an exception in case of BN? Can this be assumed for arbitrary vector functions (besides BN) as long as the parameters are differentiable? Or is BN one of the few functions where you can learn the parameters like that. I'm thinking for example about flexible ReLu's where you can learn the gradient of the linear part instead of fixing it at 1. Don't remember if Softmax had learnable parameters as well but I believe not. $\endgroup$ – evolution Nov 5 '17 at 20:18
  • $\begingroup$ Look at the answer. About the ReLU, it is not necessary to learn its slope, since the weight parameter associated with the connection from its output to the next layer plays this role. $\endgroup$ – Hossein Nov 6 '17 at 17:54
0
$\begingroup$

Your intuition is correct. You can use gradient descent to update any parameter in your network so long as you can compute the gradient of the loss function with respect to that parameter (using backpropagation). Gamma and Beta of batch normalization layers are no exceptions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.