1
$\begingroup$

I generate 100000 sample at point 1,2,3,4 and 5 so total there are 5 lac samples. I have to classify data into 2 classes. I want to normalize the data in the range of zero to one. If I normalize a single feature in all the 1 lac samples at one point such that first column or the first feature in all 1 lac samples of point 1 ${A}_1,_1 ,{ A}_2,_1 + { A}_3,_1 +....{ A}_N,_1 $ values lies from zero to one. And i repeat the same procedure for all the features. Where as in one sample there are total 16 features. But after such normalization will I not meet vanishing gradient problem? Because after this normalization first sample of point 1 contains features such that [0,0,0,0 ....0]. So during training when machine met with this sample all weights will be zero. I have doubt on my normalization method.

$\endgroup$
0
$\begingroup$

Infact the opposite, if you are normalizing with zero mean and unit variance, such normalization will help in eliminating vanishing gradient problem.

There is no reason to doubt that normalization would cause such problems.

Using batch normalization along with ReLu activation function will solve your vanishing gradient problems.

Edit: If you are normalizing in the range [0,1] for a neural network, the neurons would saturate and the training process would not move along. To avoid this, you could either normalize in range [0.1,0.9] or ideally, in range [-0.5,0.5]. The results would not differ a lot as long as you avoid saturation.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Is my normalization incorrect? $\endgroup$ – huda Nov 2 '18 at 12:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.