From this answer https://stats.stackexchange.com/a/437474/346940 seems that batch norm scales the standardized input by a factor $ \beta $... why don't we restrict this $\beta$ to be greater than zero?
if with one gradient update we pass from a $\beta > 0$ but very close to zero, to one small and negative, the input of the following layer will be flipped all of a sudden
The updates to $\beta$ are done to improve the loss. Assuming the gradient descent is working (no software bugs, the step size is small enough), then the result of $\beta$ becoming smaller and eventually negative is that the model's loss is decreasing and the model is improving.
Neural networks are compositions of many parameters, so considering a single parameter in isolation does not give the full picture of how changing it will change the network. At the same time that $\beta$ is being updated to minimize the loss, so too are all of the other parameters in the model. So it could be the case that $\beta$ is getting smaller, or even changing sign, because the weights before or after it are also changing sign or growing larger.
This observation, a weight getting larger and smaller, and possibly changing sign, is related to identifiability, because the signs and magnitudes of a single weight are not unique but instead operate in conjunction with the signs and magnitudes of the weights before and after it. See: Can we use MLE to estimate Neural Network weights? The phenomenon of is not unique to the weights and biases of the batch normalization layer. It could also happen in an ordinary neural network with several layers.
