Suppose we have a dataset with two features that are always positive for all instances. Using the two features, we want to assign the instances to one of two classes (binary classification).

To solve this task we use the following network architecture:

  • Input Layer: 2 neurons
  • Hidden Layer: 1 neuron, activation=ReLU, initialization=uniform (-0.05, 0.05)
  • Output Layer: 1 Neuron mit sigmoid, initializiaton=uniform (0.05, 0.05)

Would the hidden layer neuron be dead forever if its two weights were randomly initialized negative?

My train of thought is: If the two weights of the hidden layer neuron are randomly initialized negative, the neuron will output 0 for all instances (since all features are positive for all instances). The output of the output neuron is therefore $\sigma(0 + b)$. With $b >= 0$, all instances are assigned to class 1. With $b < 0$, all instances are assigned to class 0.


2 Answers 2


If a unit with ReLU activation gives $0$ for all the training inputs (that means also $0$ gradient based on the ReLU, which is the whole issue), it can't recover. This is dying ReLU problem, which is addressed by leaky ReLU activation. In your case, once the neuron parameters are pushed out of the data region, they're dead forever. A better way to tackle this case can be standardising your data so that it consist of both positive and negative values so that you're not left in the mercy of two unfortunate weight initialisations.

A note: It doesn't affect the concept of dying ReLU but ReLU (and all activations) takes $w^Tx+b$ as input, not just $w^Tx$. So, the neuron's output is not $0+b$.

  • $\begingroup$ Thank you for the quick answer. I edited my question. It should be: The output of the output neuron is therefore $\sigma(0+b)$ (instead of $0+b$). $\endgroup$
    – zwithouta
    Commented Feb 18, 2020 at 10:50
  • $\begingroup$ that's fine, it doesn't change the conclusion about dying neurons. $\endgroup$
    – gunes
    Commented Feb 18, 2020 at 11:35

Yes the neurons and your network would be dead forever. The same can happen when you have 2 consecutive layers with ReLU activation, even if your inputs are zero-centered.

The first layer will always output positive values. If the 2nd ReLU layer is initialized with only negative weights, it will always output 0.

The only workaround i know is to use a different activation function, such as Leaky ReLU.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.