2
$\begingroup$

I'm new to neural networks and I think I now have a good grasp of the fundamentals, but I have a question relating to normalization and activation functions.

I see places that say to normalize between -1 and 1, and some that say between 0 and 1. I also see many people recommending using the ReLU activation function for performance benefits.

I assume that the data should be normalized to suit the chosen activation function? i.e. if using ReLU then the data should be normalized between 0 and 1 as anything <0 is 0. So if I'd normalized between -1 and 1 then a big chunk of the data immediately becomes 0? If normalizing the data between -1 and 1 then I assume that'd be more suited to TANH?

Also, because ReLU is linear, could the data be normalized beyond 0-1, and maybe 0-5? Would that be advisable?

Many thanks.

$\endgroup$
  • $\begingroup$ Empirically, I observed that not normalising the data can have an effect in regression. For instance, I used ReLU activation on non-normalised data and the network was performing differently from run to run. Sometimes the regression result was quite nice, fitting well a non-linear function and sometimes the fit was lower-dimensional than the original data. My suspicion is that sometimes the data happened to be squeezed to zero by ReLU. After normalizing between -1 and 1 and using tanh things got much better. $\endgroup$ – camillejr May 21 at 11:43
1
$\begingroup$

The way the weights of hidden layers are initialized makes them expect input data with standard distribution of 0 mean and 1 variance. However Batch Normalization exists, so you don't have to wonder about the activation distribution and how it is going to change in your subsequent layers.

$\endgroup$
  • $\begingroup$ Thanks Lugi. I'd not considered that weights could also be negative. So I could normalize my input between -1 and 1, and still use ReLU? Why do some say to normalize between -1 and 1, and some say 0 and 1? $\endgroup$ – BigBadMe Feb 28 '18 at 12:06
  • 1
    $\begingroup$ Normalizing to 0 mean and 1 variance seems to be working best, and I believe it is most widely used method. Also it makes most sense: let's say you have some input feature that follows a gaussian distribution, so if you normalise it, the values in the middle of the distribution will have the least impact on the network (since they are going to be close to 0), and that's good, because we want less discerning value to have less impact. $\endgroup$ – Lugi Feb 28 '18 at 12:29
1
$\begingroup$

There's an important interplay between input normalization, weight initialization, and activation functions. The fundamental issue is that nonlinear activation functions have saturating regions, where a unit's output doesn't change (much) as its input varies. When units saturate, components of the gradient are driven to zero (or near zero). This is a consequence of the chain rule (take a look at the expressions for backprop to see why this is so). If too many units approach saturation, the gradient will become small, and learning will proceed slowly (assuming gradient-based learning rules). To speed up learning, a network should be initialized such that many units are in the linear regime, where gradients are biggest.

Various weight initialization strategies were devised to achieve these conditions. These were some of the developments that made it possible to train deeper networks. These strategies often start by assuming that inputs are distributed in a particular way (typically zero mean and unit variance for each feature). They then choose random initial weights such that the distribution of activations falls within a desirable (i.e. non-saturating) range. Of course, the initial weight distribution needed to achieve this range depends on the activation function.

Therefore, inputs should typically be normalized according to the assumptions of the weight initialization strategy, which is chosen based on the activation function.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.