Should normalization match the activation function? I'm new to neural networks and I think I now have a good grasp of the fundamentals, but I have a question relating to normalization and activation functions.
I see places that say to normalize between -1 and 1, and some that say between 0 and 1. I also see many people recommending using the ReLU activation function for performance benefits.
I assume that the data should be normalized to suit the chosen activation function? i.e. if using ReLU then the data should be normalized between 0 and 1 as anything <0 is 0. So if I'd normalized between -1 and 1 then a big chunk of the data immediately becomes 0?  If normalizing the data between -1 and 1 then I assume that'd be more suited to TANH?
Also, because ReLU is linear, could the data be normalized beyond 0-1, and maybe 0-5?  Would that be advisable?
Many thanks.
 A: The way the weights of hidden layers are initialized makes them expect input data with standard distribution of 0 mean and 1 variance. However Batch Normalization exists, so you don't have to wonder about the activation distribution and how it is going to change in your subsequent layers.
A: There's an important interplay between input normalization, weight initialization, and activation functions. The fundamental issue is that nonlinear activation functions have saturating regions, where a unit's output doesn't change (much) as its input varies. When units saturate, components of the gradient are driven to zero (or near zero). This is a consequence of the chain rule (take a look at the expressions for backprop to see why this is so). If too many units approach saturation, the gradient will become small, and learning will proceed slowly (assuming gradient-based learning rules). To speed up learning, a network should be initialized such that many units are in the linear regime, where gradients are biggest.
Various weight initialization strategies were devised to achieve these conditions. These were some of the developments that made it possible to train deeper networks. These strategies often start by assuming that inputs are distributed in a particular way (typically zero mean and unit variance for each feature). They then choose random initial weights such that the distribution of activations falls within a desirable (i.e. non-saturating) range. Of course, the initial weight distribution needed to achieve this range depends on the activation function.
Therefore, inputs should typically be normalized according to the assumptions of the weight initialization strategy, which is chosen based on the activation function.
