1
$\begingroup$

We know that batch normalization will normalize the net activations $z_n^{(l)*}$ for each layer. But I am not sure how to normalize the input of test? Here we ignore the final step of scaling and shifting in Batch Normalization.

Suppose $\{z_1^{(l)*},\cdots,z_N^{(l)*}\}$ is the net activation of entire samples under trained parameters with mean $\mu^{(l)*},$ variance $(\sigma^{(l)*})^2.$ Then for a test input $x,$ do we have:

$$x\rightarrow \hat{x}=\dfrac{x-\mu^{(0)*}}{\sigma^{(0)*}}\Rightarrow x^{(1)} \rightarrow \hat{x}^{(1)}=\dfrac{x^{(1)}-\mu^{(1)*}}{\sigma^{(1)*}}\Rightarrow x^{(2)} \rightarrow \hat{x}^{(2)}=\dfrac{x^{(2)}-\mu^{(2)*}}{\sigma^{(2)*}}\Rightarrow \cdots$$

or

only normalize at begin: $$x\rightarrow \hat{x}=\dfrac{x-\mu^{(0)*}}{\sigma^{(0)*}}$$ and no further normalizations in the later layers?

$\endgroup$

2 Answers 2

4
$\begingroup$

Batch normalization isn’t a yes-or-no decision for the entire network. You can apply it (or not) after each layer. Each layer maintains its own mean and variance; the two statistics are computed from the current batch.

With that in mind, if you include a BatchNorm layer only after the input layer, your second equation is correct. If you include a BatchNorm layer after every single layer, then your first equation is correct.

At test time, you don’t use the test set’s statistics. You use the saved statistics from the training set. But that seems orthogonal to the question you’re really asking.

$\endgroup$
2
  • $\begingroup$ So the conclusion is that, the layers normalized in test should be consistent with the layers in normalized in training. $\endgroup$ Mar 25, 2021 at 18:06
  • $\begingroup$ Yep, that’s right! $\endgroup$ Mar 25, 2021 at 21:34
0
$\begingroup$

Your first sentence is correct, batch normalization is applied at each layer. The only difference between train and test time is that at test time, the running estimates of the mean and variance are not updated.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.