5
$\begingroup$

batch normalization in a sense that in a given layer, you standardize the neurons' values, then multiply each with some trainable scaling constant, and shift them with some another trainable shifting constant.

But is it also recommended to do that in the output layer? If softmax function is used, then i don't see the point of doing it, and if not softmax is used, then is it still recommended?

$\endgroup$
2
$\begingroup$

It's probably a bad idea to apply batch norm on the last layer. I haven't seen any rigorous explanation of why, but it's probably because it introduces so much variance / randomness / regularization in the final outputs that it hurts training.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.