# Should normalization match the activation function?

I'm new to neural networks and I think I now have a good grasp of the fundamentals, but I have a question relating to normalization and activation functions.

I see places that say to normalize between -1 and 1, and some that say between 0 and 1. I also see many people recommending using the ReLU activation function for performance benefits.

I assume that the data should be normalized to suit the chosen activation function? i.e. if using ReLU then the data should be normalized between 0 and 1 as anything <0 is 0. So if I'd normalized between -1 and 1 then a big chunk of the data immediately becomes 0? If normalizing the data between -1 and 1 then I assume that'd be more suited to TANH?

Also, because ReLU is linear, could the data be normalized beyond 0-1, and maybe 0-5? Would that be advisable?

Many thanks.

• Empirically, I observed that not normalising the data can have an effect in regression. For instance, I used ReLU activation on non-normalised data and the network was performing differently from run to run. Sometimes the regression result was quite nice, fitting well a non-linear function and sometimes the fit was lower-dimensional than the original data. My suspicion is that sometimes the data happened to be squeezed to zero by ReLU. After normalizing between -1 and 1 and using tanh things got much better. – kamilazdybal May 21 '19 at 11:43