While studying Batch normalization, I came across the parameter sigma and beta in the output. And all the information said that they are added in order to retain the "expressive power of the activation function". But I did not understand what is the "expressive power". Can anyone clear my doubts?
The authors address this in the paper. Essentially, naively applying a shift & scale reduces to a network that's very close to a linear model, and linear models are a very small fraction of the general kinds of models a neural network can capture in general. In this sense, linear networks are less "expressive."
If we use scaling to give a mean of 0 and variance 1 for the inputs to a sigmoid activation, we're restricting that activation to be closer to the portion of the activation function that's approximately linear. If all layers of the network are approximately linear, then taken together, the whole network is almost linear. A linear network can be approximated by a single layer; OLS and logistic regression are special cases of neural networks with a single layer and an activation applied to that layer.
Similar phenomena can happen with other activation functions, but the nature of this will depend on the activation. For ReLU, an input following a standard normal distribution will set about half the values to 0. This might be what you want your model to do, or it might not.
If you're using a neural network, you're typically doing so because you want to use compositions of several nonlinear transformations. So we need to use a different scheme than naive scaling to ensure that batch norm doesn't destroy the network.
So, to answer your question, a neural network that is only able to model linear functions is not as expressive as a multi-layer network that is able to model a large class of non-linear functions.
From "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift" by Sergey Ioffe, Christian Szegedy:
Note that simply normalizing each input of a layer may change what the layer can represent. For instance, normalizing the inputs of a sigmoid would constrain them to the linear regime of the nonlinearity. To address this, we make sure that the transformation inserted in the network can represent the identity transform. To accomplish this, we introduce, for each activation $x^{(k)}$, a pair of parameters $\gamma^{(k)}, \beta^{(k)}$, which scale and shift the normalized value: $$ y^{(k)} = \gamma^{(k)}\hat{x}^{(k)} + \beta^{(k)}. $$ These parameters are learned along with the original model parameters, and restore the representation power of the network. Indeed, by setting $\gamma^{(k)} = \sqrt{\text{Var}\left[x^{(k)}\right]}$ and $\beta^{(k)} = \mathbb{E}\left[x^{(k)}\right]$, we could recover the original activations, if that were the optimal thing to do.
Some more reading:
