How to implement Batch Norm to Deep learning Neural Networks?

Question

I'm studying at coursea.com Neural Networks with deep learning course. I have a problem with implementing A Batch Norm to Mini-Batch Gradient descent. More accurately, in gamma and beta hyper-parameters. As Andrew Ng said, they are learnable params and they are modified in gradient descent as weights,bias,etc. My question is: how should I initialize the Gamma and Beta -> Randomly or there are some default values and in which shape they will be? and should I initialize them where I do it with weights or bias or in forward prop?

Answer 1

In PyTorch and Keras, the gamma and beta parameters are initialized as a vector of ones and zero, respectively when the model is constructed. Initializing beta as zero intuitively makes sense -- it's basically a BatchNorm specific bias term -- and gamma acts as a scaling operation on the pre-output BatchNorm vector, so initializing it as ones implies that we initially do not scale at all and only learn to do so if appropriate.

To calculate the derivates of BN parameters $\gamma$ and $\beta$, we can write that the output of BN $y$ for a given batch of examples $x$ is $y = \gamma \hat{x} + \beta$, where $\hat{x} = \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}}$. $\mu$ and $\sigma$ are calculated on the batch; $\epsilon$ is just a small number to avoid division by zero. Computation follows the same form as any backprop derivative computation. $\frac{\partial y}{\partial \gamma}$ is clearly $\hat{x}$; $\frac{\partial y}{\partial \beta}$ is 1.

Now, suppose the final output of the model is some variable $z$. In the course of backprop, say that we've calculated derivates up to $\frac{\partial z}{\partial y}$. Then, $\frac{\partial z}{\partial \gamma} = \frac{\partial z}{\partial y}\cdot\hat{x}$, and $\frac{\partial z}{\partial \beta} = \frac{\partial z}{\partial y}$. You can verify this using the chain rule.

Then, your ultimate update rules are:

$$\gamma \leftarrow \gamma - \eta * \frac{\partial z}{\partial y}\cdot\hat{x}$$ $$\beta \leftarrow \beta - \eta * \frac{\partial z}{\partial y}$$