1
$\begingroup$

For Neural Networks with ReLU activations it is common practice to use the initalization introduced by He et al. to keep the variance of layer activations constant throughout the network.

On the other hand, for a classification task, this course suggests that a good initialization should result in the expected loss for random guessing, e.g ln(number of classes) for a softmax classifier.

These two criteria are not compatible in the network I am training: When using He initialization, the loss before training is much higher. The necessary variance to arrive at the expected loss is much smaller than sqrt(2/N).

With both initializations I am able to train the network and they reach similar losses after training. However, the learned filters are quite different: For the initialization with the smaller variance, in some of the filters all weights get the same, detecting just how light or dark a patch is, which seems odd. With He, init on the other hand, the loss (which already starts higher) makes a big jump up in the first step before drastically decreasing in the next. This jump can be prevented by smaller learning rates which however slow the whole convergence afterwards.

Why is the loss with He initalization higher than random guessing? Which criterion should one prioritize? Each class in my dataset has the same number of examples. I use no kind of regularization.

$\endgroup$
2
  • $\begingroup$ are the inputs properly normalized? $\endgroup$
    – shimao
    Jun 12, 2018 at 14:51
  • $\begingroup$ @shimao Yes, I subtracted the mean and divided by the standard deviation. $\endgroup$
    – McLawrence
    Jun 12, 2018 at 14:56

1 Answer 1

0
$\begingroup$

Without actually seeing the code and the problem - most probably there's a problem with your setup. He init, or any other random init for this matter, sets the weights to random numbers around $0$, so the outputs should be completely random. If you're using a cross-entropy loss, and your outputs are completely random, you should get a loss in the vicinity of $-\log(1/C)=\log(C)$, where $C$ is the number of classes.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.