I'm studying at coursea.com Neural Networks with deep learning course. I have a problem with implementing A Batch Norm to Mini-Batch Gradient descent. More accurately, in gamma and beta hyper-parameters. As Andrew Ng said, they are learnable params and they are modified in gradient descent as weights,bias,etc. My question is: how should I initialize the Gamma and Beta -> Randomly or there are some default values and in which shape they will be? and should I initialize them where I do it with weights or bias or in forward prop?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
14
In PyTorch and Keras, the gamma and beta parameters are initialized as a vector of ones and zero, respectively when the model is constructed. Initializing beta as zero intuitively makes sense -- it's basically a BatchNorm specific bias term -- and gamma acts as a scaling operation on the pre-output BatchNorm vector, so initializing it as ones implies that we initially do not scale at all and only learn to do so if appropriate.
To calculate the derivates of BN parameters $\gamma$ and $\beta$, we can write that the output of BN $y$ for a given batch of examples $x$ is $y = \gamma \hat{x} + \beta$, where $\hat{x} = \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}}$. $\mu$ and $\sigma$ are calculated on the batch; $\epsilon$ is just a small number to avoid division by zero. Computation follows the same form as any backprop derivative computation. $\frac{\partial y}{\partial \gamma}$ is clearly $\hat{x}$; $\frac{\partial y}{\partial \beta}$ is 1.
Now, suppose the final output of the model is some variable $z$. In the course of backprop, say that we've calculated derivates up to $\frac{\partial z}{\partial y}$. Then, $\frac{\partial z}{\partial \gamma} = \frac{\partial z}{\partial y}\cdot\hat{x}$, and $\frac{\partial z}{\partial \beta} = \frac{\partial z}{\partial y}$. You can verify this using the chain rule.
Then, your ultimate update rules are:
$$\gamma \leftarrow \gamma - \eta * \frac{\partial z}{\partial y}\cdot\hat{x}$$ $$\beta \leftarrow \beta - \eta * \frac{\partial z}{\partial y}$$
-
$\begingroup$ Is BN used in output or in hidden layers only? $\endgroup$ Mar 30, 2020 at 11:59
-
$\begingroup$ And how compute dervatives of Gamma and Beta $\endgroup$ Mar 30, 2020 at 11:59
-
$\begingroup$ It would be weird to use BN in the output layer, since 1) BN serves to ensure that the distribution of data from layer-to-layer is approximately the same (dealing w/ internal covariate shift), and that 2) you want to make a classification based on the activations of the output layer, so it would be strange to normalize that before classification. The derivates of gamma and beta are computed in exactly the same way as any other derivate in backprop. I'll edit my answer with some computations. $\endgroup$ Mar 30, 2020 at 20:41
-
-
$\begingroup$ I have one more question? -> if I got n-vecto of beta and gamma for each layer -> Should I in : $\endgroup$ Mar 30, 2020 at 22:37