13
$\begingroup$

In most neural networks that I've seen, especially CNNs, a commonality has been the lack of batch normalization just before the last fully connected layer. So usually there's a final pooling layer, which immediately connects to a fully connected layer, and then to an output layer of categories or regression. I can't find it now but, I remember seeing a vague reference on this that concluded batch normalization before the last FC layer didn't make much of a difference. If this is true, why is this?

In practice, it seems like the last FC layer tends to have around 10% of its neurons dead for any given input (although, I haven't measured neuron contiguity). This proportion tends to grow considerably when you increase the FC layer, especially when starting from previously pre-trained models.

$\endgroup$
5
  • $\begingroup$ The accepted answer is poorly-reasoned; do you have specific architectures in mind? My best guess is, you aren't seeing BN after a pooling layer, to which there is a far more relevant explanation than the answer's. $\endgroup$ Nov 14, 2019 at 3:20
  • $\begingroup$ @OverLordGoldDragon: I agree that the current answer is a bit lacking, but I don’t otherwise have a better explanation. You’re definitely right that I haven’t seen it after global looking layers either. If you wanted to provide a more thorough answer I would be thrilled. $\endgroup$
    – Alex R.
    Nov 14, 2019 at 18:25
  • $\begingroup$ Sounds good - I'll cook up an answer $\endgroup$ Nov 14, 2019 at 18:37
  • $\begingroup$ I never found time for this but can direct you to a discussion with references I had in mind. $\endgroup$ Dec 7, 2020 at 17:58
  • 1
    $\begingroup$ To put things in simple manner, see it like this: the more normalize (or partial values in case of activation like ReLU) is observed by end layer the less efficient decision making ability is, how can the last layer decide if something is good or bad or relevant if you are already providing modified or clipped or normalized data towards the end, which would further affect the network's ability to move values and help softmax (for example) scale further to pick one. the network must be able to move from generic to specific understanding as you move towards the end. $\endgroup$
    – MANU
    Dec 4, 2021 at 13:02

1 Answer 1

11
$\begingroup$

I am pretty sure that batch norm before the last FC layer not only does not help, but it hurts performance pretty severely.

My intuition is that the network has to learn a representation which is mostly invariant to the stochasticity inherent in batch norm. At the same time, by the time it reaches the last layer, it has to convert that representation back into a fairly precise prediction. It's likely that a single FC layer is not powerful enough to perform that conversion.

Another way to say it is that batch norm (like dropout) adds stochasticity to the network, and the network learns to be robust to this stochasticity. However it's simply impossible for the network to cope with stochasticity right before the output.

$\endgroup$
7
  • 2
    $\begingroup$ I can confirm that I just observed this behavior - the network would not learn with batch norm before the last layer, and worked great after removing that specific batch norm operation (but leaving batch norm elsewhere). $\endgroup$
    – zplizzi
    Sep 29, 2019 at 20:30
  • $\begingroup$ @zplizzi Between which exact layers had you placed BN? $\endgroup$ Nov 14, 2019 at 3:23
  • $\begingroup$ @OverLordGoldDragon honestly I don't remember now, although I'm pretty certain I didn't have any pooling layers in the network at all. $\endgroup$
    – zplizzi
    Nov 14, 2019 at 18:26
  • $\begingroup$ I actually think this might be wrong in situations where there is class imbalance, as I’ve seen class-specific-dropout help quite well on the last layer. $\endgroup$
    – Alex R.
    Nov 14, 2019 at 18:27
  • 1
    $\begingroup$ @OverLordGoldDragon I'm also wondering what class-specific dropout means, but I've run into pieces of work where they use batch normalization to do Domain-Specific Batch Normalization for Unsupervised Domain Adaptation (using the same network to process data from different domains) and to make efficient use of Adversarial Examples (to) Improve Image Recognition. $\endgroup$ Dec 7, 2020 at 13:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.