5
$\begingroup$

In most neural networks that I've seen, especially CNNs, a commonality has been the lack of batch normalization just before the last fully connected layer. So usually there's a final pooling layer, which immediately connects to a fully connected layer, and then to an output layer of categories or regression. I can't find it now but, I remember seeing a vague reference on this that concluded batch normalization before the last FC layer didn't make much of a difference. If this is true, why is this?

In practice, it seems like the last FC layer tends to have around 10% of its neurons dead for any given input (although, I haven't measured neuron contiguity). This proportion tends to grow considerably when you increase the FC layer, especially when starting from previously pre-trained models.

$\endgroup$
  • $\begingroup$ The accepted answer is poorly-reasoned; do you have specific architectures in mind? My best guess is, you aren't seeing BN after a pooling layer, to which there is a far more relevant explanation than the answer's. $\endgroup$ – OverLordGoldDragon Nov 14 '19 at 3:20
  • $\begingroup$ @OverLordGoldDragon: I agree that the current answer is a bit lacking, but I don’t otherwise have a better explanation. You’re definitely right that I haven’t seen it after global looking layers either. If you wanted to provide a more thorough answer I would be thrilled. $\endgroup$ – Alex R. Nov 14 '19 at 18:25
  • $\begingroup$ Sounds good - I'll cook up an answer $\endgroup$ – OverLordGoldDragon Nov 14 '19 at 18:37
2
$\begingroup$

I am pretty sure that batch norm before the last FC layer not only does not help, but it hurts performance pretty severely.

My intuition is that the network has to learn a representation which is mostly invariant to the stochasticity inherent in batch norm. At the same time, by the time it reaches the last layer, it has to convert that representation back into a fairly precise prediction. It's likely that a single FC layer is not powerful enough to perform that conversion.

Another way to say it is that batch norm (like dropout) adds stochasticity to the network, and the network learns to be robust to this stochasticity. However it's simply impossible for the network to cope with stochasticity right before the output.

$\endgroup$
  • $\begingroup$ I can confirm that I just observed this behavior - the network would not learn with batch norm before the last layer, and worked great after removing that specific batch norm operation (but leaving batch norm elsewhere). $\endgroup$ – zplizzi Sep 29 '19 at 20:30
  • $\begingroup$ @zplizzi Between which exact layers had you placed BN? $\endgroup$ – OverLordGoldDragon Nov 14 '19 at 3:23
  • $\begingroup$ @OverLordGoldDragon honestly I don't remember now, although I'm pretty certain I didn't have any pooling layers in the network at all. $\endgroup$ – zplizzi Nov 14 '19 at 18:26
  • $\begingroup$ I actually think this might be wrong in situations where there is class imbalance, as I’ve seen class-specific-dropout help quite well on the last layer. $\endgroup$ – Alex R. Nov 14 '19 at 18:27
  • $\begingroup$ @AlexR. What is "class-specific dropout"? That'd involve adjusting masks per-sample, unsure how that'd work or whether a bias would be introduced - have a reference? $\endgroup$ – OverLordGoldDragon Nov 14 '19 at 18:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.