1
$\begingroup$

In their Fixup Initialization: Residual Learning Without Normalization, authors suggest:

Fixup initialization (or: How to train a deep residual network without normalization)
1. Initialize the classification layer and the last layer of each residual branch to 0.
2. Initialize every other layer using a standard method (e.g., Kaiming He), and scale only the weight layers inside residual branches by ... .
3. Add a scalar multiplier (initialized at 1) in every branch and a scalar bias (initialized at 0) before each convolution, linear, and element-wise activation layer.

I used to think that zero init is a pitfall. What am I missing?

a cutout from Figure 1

$\endgroup$
1
$\begingroup$

Zeros initialization is a pitfall, but as far a i see and someone can correct me if i am wrong.

Here they are only initializing the second conv filter with zeros, with multiple modifications like adding a bias before the conv and rescaling the first conv by the position of the layer.So all these together help it avoid the the vanilla zero initialization.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.