Although a Relu activation function can deal with real value number but I have tried scaling the dataset in the range [0,1] (min-max scaling) is more effective before feed it to the neural network. on the other hand, the batch normalization (BN) is also normalizing data before passed to the non-linearity layer (activation function). I was wondering if the min-max scaling is still needed when BN is applied. can we perform min-max scaling and BN together?. It would be nice if someone guides me to the better understanding
$\begingroup$
$\endgroup$
1
asked Dec 3, 2016 at 3:03
-
$\begingroup$ I think you can let go of batch normalization. It is not needed any more. $\endgroup$– Souradeep NandaCommented Dec 5, 2016 at 5:39
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
As mentioned, it's best to use [-1, 1] min-max scaling or zero-mean, unit-variance standardization. Scaling your data into [0, 1] will result in slow learning.
To answer your question: Yes, you should still standardize your inputs to a network that uses Batch Normalization. This will ensure that inputs to the first layer have zero mean and come from the same distribution, while Batch Normalization on subsequent layers will ensure that inputs to those layers have zero mean in expectation and that their distributions do not drift over time.
The reasons that we want zero mean and stable input distribution are discussed further in Section 4.3 of Efficient BackProp.
-
5$\begingroup$ As a hack, can I use batch norm in the first layer of my network? I am feeling lazy to preprocess the input :P $\endgroup$ Commented Mar 15, 2020 at 7:04
-
2$\begingroup$ Yes, see the "Alternate to Data Preparation" section in machinelearningmastery.com/…. And if I remember correctly, Jeremy Howard also mentions this technique in the first edition of his Practical Deep Learning for Coders video series. $\endgroup$– ImranCommented Jun 23, 2020 at 21:58
$\begingroup$
$\endgroup$
In this case scaling data would only influence the first layer of your network. Also if you are scaling your input it's better to scale it to [-1, 1], but it's best to scale it to 0 mean and 1 variance (since your weights are probably initialized to expect such distribution).
Not that it's going to make a huge difference anyway.
