In their Fixup Initialization: Residual Learning Without Normalization, authors suggest:

Fixup initialization (or: How to train a deep residual network without normalization)
1. Initialize the classification layer and the last layer of each residual branch to 0.
2. Initialize every other layer using a standard method (e.g., Kaiming He), and scale only the weight layers inside residual branches by ... .
3. Add a scalar multiplier (initialized at 1) in every branch and a scalar bias (initialized at 0) before each convolution, linear, and element-wise activation layer.

I used to think that zero init is a pitfall. What am I missing?

a cutout from Figure 1


Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.