1
$\begingroup$

I was looking through labml's implementation of StyleGan2, and came across two normalisations:

in the DiscriminatorBlock:

"Scaling factor $\frac{1}{\sqrt 2}$ after adding the residual"

 self.scale = 1 / math.sqrt(2)

in the PathLengthPenalty:

"Calculate (g(w)⋅y) and normalize by the square root of image size. This is scaling is not mentioned in the paper but was present in their implementation."

output = (x * y).sum() / math.sqrt(image_size)

as noted, there is no explanation for the necessity of them in the original paper, nor in labml's explanations - which raises the following questions:

  1. why are those normalisations needed?
  2. why those specific normalisations where chosen, in each case?
  3. how one would generally know which normalisations to choose in a deep learning project?

I would appreciate any helpful tips, ideas and insights on this matter. thanks.

$\endgroup$

1 Answer 1

1
$\begingroup$

I don't no the real answer if there is any.

I'll try to answer on the square root normalization. Lets say that we have two independent random variable, x and y. Both have the variance. What would be the variance of their sum?

Var(x + y) = Var(x) + Var(y) = 2*const.

Now, lets define another two random variables: x' = x/sqrt(2), y' = y/sqrt(2). What would be the variance of their sum?

Var(x' + y') = Var(x') + Var(y') = Var(x) / 2 + Var(y) / 2 = const.

How does it relates to residual connections?

Generally speaking, we want to prevent from gradients to explode. Try to think about x and y as a random tensors (y will be the residualconnection), whose variance is equal. When you sum them up (applying the residual connection) you increase the variance. Now, if do it layer by layer I guess you will end up incresing gradient's magnitude. So scaling down by square root make sense to me from the point of view of variance preservation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.