Suppose we have a dataset with two features that are always positive for all instances. Using the two features, we want to assign the instances to one of two classes (binary classification).

To solve this task we use the following network architecture:

  • Input Layer: 2 neurons
  • Hidden Layer: 1 neuron, activation=ReLU, initialization=uniform (-0.05, 0.05)
  • Output Layer: 1 Neuron mit sigmoid, initializiaton=uniform (0.05, 0.05)

Would the hidden layer neuron be dead forever if its two weights were randomly initialized negative?

My train of thought is: If the two weights of the hidden layer neuron are randomly initialized negative, the neuron will output 0 for all instances (since all features are positive for all instances). The output of the output neuron is therefore $\sigma(0 + b)$. With $b >= 0$, all instances are assigned to class 1. With $b < 0$, all instances are assigned to class 0.


If a unit with ReLU activation gives $0$ for all the training inputs (that means also $0$ gradient based on the ReLU, which is the whole issue), it can't recover. This is dying ReLU problem, which is addressed by leaky ReLU activation. In your case, once the neuron parameters are pushed out of the data region, they're dead forever. A better way to tackle this case can be standardising your data so that it consist of both positive and negative values so that you're not left in the mercy of two unfortunate weight initialisations.

A note: It doesn't affect the concept of dying ReLU but ReLU (and all activations) takes $w^Tx+b$ as input, not just $w^Tx$. So, the neuron's output is not $0+b$.

  • $\begingroup$ Thank you for the quick answer. I edited my question. It should be: The output of the output neuron is therefore $\sigma(0+b)$ (instead of $0+b$). $\endgroup$ – zwithouta Feb 18 '20 at 10:50
  • $\begingroup$ that's fine, it doesn't change the conclusion about dying neurons. $\endgroup$ – gunes Feb 18 '20 at 11:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.