1
$\begingroup$

From the paper by Ioffe, it is mentioned batch norm's benefit is that it is able to fix the distribution of the layers' input, which allow the layers to learn the same representation rather than having to constantly compensate their learning due to a varying distributions. However, as this could potentially misrepresent or change the representation of this distribution following normalization, they added a gamma and beta variable to recompensate for this normalization.

In that case, since the gamma and beta are different for all activations and layers, wouldn't that make the distribution not fixed again? I.e. It is not that all layers will have Gaussian distributions of their input, but rather somewhere close to Gaussian given the linear transformation from using gamma and beta.

So in that case, batch normalization is not essentially giving a fixed distribution to all layer inputs, but rather a slightly varying distribution that is near Gaussian distributions. Is this correct?

This leads to the next question: why Gaussian and not other forms of distributions? Although arguably Gaussian appears most commonly in nature, but I would suspect for different sets of data, or perhaps the more specialized data, certain distributions could be more optimal.

$\endgroup$
1
$\begingroup$

It's okay for different layers to have different input distributions. The key thing is that an individual layer always sees the same input distribution. That way it doesn't have to compensate for the changing output distributions of previous layers.

You could use a different non-Gaussian distribution if you want but the nice thing about Gaussian distributions is that they are parameterized by just two values: the mean and the variance.

$\endgroup$
0
$\begingroup$

I think that the reason for assuming a gaussian distribution has to do with the fact that a linear combination of random variables is, by the central limit theorem, gaussian. So after the linear combination you apply a standard normalization so the distribution remain fix. Remark this cannot be done after non-linearities (as example) because they automatically change distribution and you cannot achieve a fix distribution, that is the desired property (avoid changes in distribution to input layer)

You then let the free parameters to adjust the mean and variance (of and ALWAYS gaussian distribution) because, either you add or you do not add this parameters, a linear transformation does not change the shape of a distribution. This is a way to give some freedom in case the standard distribution is not the best you need. What you basically do in batch norm is make the input of all layers have the same shape in the same range. It is something similar to an input normalization. When you normalize the input you normally achieve better results and fast training.

Hope it helps.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.