3
$\begingroup$

In their paper Group Normalization the author introduce GroupNorm(GN) as a replacement for BatchNorm. They show that LayerNorm(LN) and InstanceNorm(IN) are extreme cases of GN. They also show that GN significantly outperforms LN and IN. My experiments also confirm this.
But I didn't find an intuition as to why that is the case. Why does GN which is the middle ground between LN and IN, outperforms them?

enter image description here

$\endgroup$

1 Answer 1

1
$\begingroup$

I don't think there's any formal reasoning as to why GN would outperform both LN and IN. So this is all based on my intuition.

The difference in LN, GN, and IN is essentially the number of groups you use to perform normalization over channel features. For LN, the number of groups is 1; and for IN, the number of groups is the number of input channels.

So in some sense, LN may not sufficiently allow for separated normalization of different "kinds" of channel features (say some groups of channels are dedicated for edge features, and some are for bulk features). On the other hand, IN does not allow any grouping of features, and implicitly assumes that every channel represents a distinct feature that has to be separately normalized.

In reality, I guess there is some underlying natural "grouping" of channel features that is between the two extremes, and GN takes advantage of this.

$\endgroup$
1
  • $\begingroup$ Interesting hypothesis. It would be interesting to see how features learned by layers in the same group are related to each other, and how they are different from features learned by layers in other groups. $\endgroup$
    – Sia Rezaei
    Commented May 9, 2022 at 23:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.