In general should I retrain my neural network with fewer neurons so that it has fewer dead ReLU neurons? I've read conflicting opinions about dead ReLUs. Some sources say dead ReLUs are good because they encourage sparsity. Others say they're bad because dead ReLUs are dead forever and inhibit learning. Is there a happy medium?


There's a difference between dead ReLUs and ReLUs that are silent on many--but not all--inputs. Dead ReLUs are to be avoided, whereas mostly-silent ReLUs can be useful because of the sparsity they induce.

Dead ReLUs have entered a parameter regime where they're always in the negative domain of the activation function. This could happen, for example, if the bias is set to a large negative value. Because the activation function is zero for negative values, these units are silent for all inputs. When a ReLU is silent, the gradient of the loss function with respect to the parameters is zero, so no parameter updates will occur with gradient-based learning. Because dead ReLUs are silent for all inputs, they're trapped in this regime.

Contrast this with a ReLU that's silent on many but not all inputs. In this case, the gradient is still zero when the unit is silent. If we're using an online learning procedure like minibatch/stochastic gradient descent, no parameter updates will occur for inputs that cause the unit to be silent. But, updates are still possible for other inputs, where the unit is active and the gradient is nonzero.

Because dead ReLUs are silent for all inputs, they contribute nothing to the network, and are wasted. From an information theoretic perspective, any unit that has the same output value for all inputs (whether zero or not) carries no information about the input. Mostly-silent ReLUs behave differently for different inputs, and therefore maintain the ability to carry useful information.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.