When training my CNN, I notice that after several SGD updates, some weights of the layers do not change any more. Is this a normal situation? Will all the weights of the network layers change during each training iteration?

  • $\begingroup$ How did you find out that they are not changing? Could it simply be that the changes are very small? Is it always the same weights not updating, or a different set of weights at each step? $\endgroup$ – Jan Kukacka Feb 6 '18 at 10:38
  • $\begingroup$ @JanKukacka: I saved the network weights every 20000 iterations. By looking back the saved weights, I found that some of the weights stopped changing after several iterations. I am trying different network structures and for different structures, the stop-changing weights are not always the same. $\endgroup$ – madnerd Feb 6 '18 at 10:56

This can happen with ReLU activation functions, when a conv filter before the activation gets to a state when it always outputs a negative number and the following ReLU truncates the output, thus preventing any gradient propagation. Generally, CNNs are less prone to this problem (as the weights are shared between multiple locations), but it can still happen. A large network can usually deal with the fact that some of the units are "dead", however, you might try some other activation function such as Leaky ReLU or PReLU to avoid this problem.

  • $\begingroup$ Do these "dead" units affect the model's performance? I found this paper saying that Leaky ReLU performs almost identical to ReLU. $\endgroup$ – madnerd Feb 6 '18 at 12:56
  • $\begingroup$ Dead neurons shouldn't be a concern if you've already reached your target performance. If you're interested in weight reduction, you can prune these dead weights to save memory space. As Jan suggested, you can use Leaky ReLU (or) PReLU to address this issue. While doing so, you can observe that the number of dead neurons will be less compared to using ReLU. $\endgroup$ – Avis Feb 7 '18 at 2:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.