1
$\begingroup$

References:

I will use pseudo TensorFlow-like code to be very specific about the tensor axes. I assume an input tensor x of shape [B,T,F], where B is the batch-dim, T is the time-dim, and F is the feature-dim. (The GN paper uses a tensor of shape [N,C,H,W], where N==B, C==F, and H is height and W is width. Width/height are the spatial axes, which correspond to our single axis T). I will also refer to the specific axes by B/T/F.

Batch normalization

In BN, you calculate:

mean = reduce_mean(x, axes=[B,T])  # shape [F] or [1,1,F]

As this is done over B/T, to keep the statistics more stable, running statistics over the whole train corpora are collected.

Layer normalization

In LN, you calculate:

mean = reduce_mean(x, axes=[F])  # shape [B,T] or [B,T,1]

As this is independent for every frame, no running statistics are needed.

Group normalization

In GN, you calculate:

F_ = F//G
x_ = reshape(x, [B, T, G, F_])
mean = reduce_mean(x_, axes=[T, F_])  # shape [B,G] or [B,1,G,1]

It normalizes over all axes except batch and G.

Note, the TF code in the GN paper (Fig 3) is:

def GroupNorm(x, gamma, beta, G, eps=1e−5):
  # x: input features with shape [N,C,H,W]
  # gamma, beta: scale and offset, with shape [1,C,1,1]
  # G: number of groups for GN
  N, C, H, W = x.shape
  x = tf.reshape(x, [N, G, C // G, H, W])
  mean, var = tf.nn.moments(x, [2, 3, 4], keep dims=True)  # shape [N,G,1,1,1]
  x = (x − mean) / tf.sqrt(var + eps)
  x = tf.reshape(x, [N, C, H, W])
  return x ∗ gamma + beta

In the GN paper, it states that GN with G=1 is equivalent to LN. But I don't see that? The mean is calculated different in each case.

Or maybe the interpretation is actually N=B*T, and W=H=1? In that case, it is equivalent. But this would be very misleading and confusing. Esp from Fig 2 of the GN paper. In the LN paper, there is no H/W axis, or at least it's not relevant. Only the hidden dimension is (i.e. channel axis, or feature axis).

Fig 3 from the GN paper is also misleading (also here): In this figure, it looks like layer-normalization normalizes over H/W as well. But this is not the case (at least commonly, and also with the default options in common frameworks like TF or PyTorch). This figure matches though the default behavior for group-normalization as it is implemented in common frameworks (like TFA or PyTorch).

The same (wrong?) statement about GN with G=1 equivalence to LN is also in the TensorFlow Addons (TFA) documentation. However, looking at the code of TFA and also PyTorch, it seems not to be the case (for the common default arguments).

I also posted a GitHub issue for TFA here.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.