19
$\begingroup$

In most neural networks that I've seen, especially CNNs, a commonality has been the lack of batch normalization just before the last fully connected layer. So usually there's a final pooling layer, which immediately connects to a fully connected layer, and then to an output layer of categories or regression. I can't find it now but, I remember seeing a vague reference on this that concluded batch normalization before the last FC layer didn't make much of a difference. If this is true, why is this?

In practice, it seems like the last FC layer tends to have around 10% of its neurons dead for any given input (although, I haven't measured neuron contiguity). This proportion tends to grow considerably when you increase the FC layer, especially when starting from previously pre-trained models.

Improve this question
$\endgroup$
5
  • $\begingroup$ The accepted answer is poorly-reasoned; do you have specific architectures in mind? My best guess is, you aren't seeing BN after a pooling layer, to which there is a far more relevant explanation than the answer's. $\endgroup$
    – OverLordGoldDragon
    Commented Nov 14, 2019 at 3:20
  • $\begingroup$ @OverLordGoldDragon: I agree that the current answer is a bit lacking, but I don’t otherwise have a better explanation. You’re definitely right that I haven’t seen it after global looking layers either. If you wanted to provide a more thorough answer I would be thrilled. $\endgroup$
    – Alex R.
    Commented Nov 14, 2019 at 18:25
  • $\begingroup$ Sounds good - I'll cook up an answer $\endgroup$
    – OverLordGoldDragon
    Commented Nov 14, 2019 at 18:37
  • $\begingroup$ I never found time for this but can direct you to a discussion with references I had in mind. $\endgroup$
    – OverLordGoldDragon
    Commented Dec 7, 2020 at 17:58
  • 1
    $\begingroup$ To put things in simple manner, see it like this: the more normalize (or partial values in case of activation like ReLU) is observed by end layer the less efficient decision making ability is, how can the last layer decide if something is good or bad or relevant if you are already providing modified or clipped or normalized data towards the end, which would further affect the network's ability to move values and help softmax (for example) scale further to pick one. the network must be able to move from generic to specific understanding as you move towards the end. $\endgroup$
    – MANU
    Commented Dec 4, 2021 at 13:02

2 Answers 2

Reset to default
14
$\begingroup$

I am pretty sure that batch norm before the last FC layer not only does not help, but it hurts performance pretty severely.

My intuition is that the network has to learn a representation which is mostly invariant to the stochasticity inherent in batch norm. At the same time, by the time it reaches the last layer, it has to convert that representation back into a fairly precise prediction. It's likely that a single FC layer is not powerful enough to perform that conversion.

Another way to say it is that batch norm (like dropout) adds stochasticity to the network, and the network learns to be robust to this stochasticity. However it's simply impossible for the network to cope with stochasticity right before the output.

Improve this answer
$\endgroup$
7
  • 3
    $\begingroup$ I can confirm that I just observed this behavior - the network would not learn with batch norm before the last layer, and worked great after removing that specific batch norm operation (but leaving batch norm elsewhere). $\endgroup$
    – zplizzi
    Commented Sep 29, 2019 at 20:30
  • $\begingroup$ @zplizzi Between which exact layers had you placed BN? $\endgroup$
    – OverLordGoldDragon
    Commented Nov 14, 2019 at 3:23
  • $\begingroup$ @OverLordGoldDragon honestly I don't remember now, although I'm pretty certain I didn't have any pooling layers in the network at all. $\endgroup$
    – zplizzi
    Commented Nov 14, 2019 at 18:26
  • $\begingroup$ I actually think this might be wrong in situations where there is class imbalance, as I’ve seen class-specific-dropout help quite well on the last layer. $\endgroup$
    – Alex R.
    Commented Nov 14, 2019 at 18:27
  • 1
    $\begingroup$ @OverLordGoldDragon I'm also wondering what class-specific dropout means, but I've run into pieces of work where they use batch normalization to do Domain-Specific Batch Normalization for Unsupervised Domain Adaptation (using the same network to process data from different domains) and to make efficient use of Adversarial Examples (to) Improve Image Recognition. $\endgroup$
    – HelloGoodbye
    Commented Dec 7, 2020 at 13:47
0
$\begingroup$

I hope this intuition helps:

A separation based on the layer's functionality: The hidden layers are extracting the features, but the last layer is using extracted features to decide.

To extract the right features we face the covariate shift problem where we don't want the change in input distribution affect the extracted features (like when training data is all black cats and test data is cats from all colors, for the cat/non-cat classification).

But for the last layer the covariate shift is dealt with by previous layers when extracting the features, and we don't want to mess with the way it is predicting the output.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.