Skip to main content
Tweeted twitter.com/StackStats/status/1046097620601581569
added 301 characters in body
Source Link
Alex R.
  • 14.1k
  • 4
  • 29
  • 51

In most neural networks that I've seen, especially CNNs, a commonality has been the lack of batch normalization just before the last fully connected layer. So usually there's a final pooling layer, which immediately connects to a fully connected layer, and then to an output layer of categories or regression. I can't find it now but, I remember seeing a vague reference on this that concluded batch normalization before the last FC layer didn't make much of a difference. If this is true, why is this?

In practice, it seems like the last FC layer tends to have around 10% of its neurons dead for any given input (although, I haven't measured neuron contiguity). This proportion tends to grow considerably when you increase the FC layer, especially when starting from previously pre-trained models.

In most neural networks that I've seen, especially CNNs, a commonality has been the lack of batch normalization just before the last fully connected layer. So usually there's a final pooling layer, which immediately connects to a fully connected layer, and then to an output layer of categories or regression. I can't find it now but, I remember seeing a vague reference on this that concluded batch normalization before the last FC layer didn't make much of a difference. If this is true, why is this?

In most neural networks that I've seen, especially CNNs, a commonality has been the lack of batch normalization just before the last fully connected layer. So usually there's a final pooling layer, which immediately connects to a fully connected layer, and then to an output layer of categories or regression. I can't find it now but, I remember seeing a vague reference on this that concluded batch normalization before the last FC layer didn't make much of a difference. If this is true, why is this?

In practice, it seems like the last FC layer tends to have around 10% of its neurons dead for any given input (although, I haven't measured neuron contiguity). This proportion tends to grow considerably when you increase the FC layer, especially when starting from previously pre-trained models.

Source Link
Alex R.
  • 14.1k
  • 4
  • 29
  • 51

Lack of Batch Normalization Before Last Fully Connected Layer

In most neural networks that I've seen, especially CNNs, a commonality has been the lack of batch normalization just before the last fully connected layer. So usually there's a final pooling layer, which immediately connects to a fully connected layer, and then to an output layer of categories or regression. I can't find it now but, I remember seeing a vague reference on this that concluded batch normalization before the last FC layer didn't make much of a difference. If this is true, why is this?