From the paper by Ioffe, it is mentioned batch norm's benefit is that it is able to fix the distribution of the layers' input, which allow the layers to learn the same representation rather than having to constantly compensate their learning due to a varying distributions. However, as this could potentially misrepresent or change the representation of this distribution following normalization, they added a gamma and beta variable to recompensate for this normalization.
In that case, since the gamma and beta are different for all activations and layers, wouldn't that make the distribution not fixed again? I.e. It is not that all layers will have Gaussian distributions of their input, but rather somewhere close to Gaussian given the linear transformation from using gamma and beta.
So in that case, batch normalization is not essentially giving a fixed distribution to all layer inputs, but rather a slightly varying distribution that is near Gaussian distributions. Is this correct?
This leads to the next question: why Gaussian and not other forms of distributions? Although arguably Gaussian appears most commonly in nature, but I would suspect for different sets of data, or perhaps the more specialized data, certain distributions could be more optimal.