Suppose we have a dataset with two features that are always positive for all instances. Using the two features, we want to assign the instances to one of two classes (binary classification).
To solve this task we use the following network architecture:
- Input Layer: 2 neurons
- Hidden Layer: 1 neuron, activation=ReLU, initialization=uniform (-0.05, 0.05)
- Output Layer: 1 Neuron mit sigmoid, initializiaton=uniform (0.05, 0.05)
Would the hidden layer neuron be dead forever if its two weights were randomly initialized negative?
My train of thought is: If the two weights of the hidden layer neuron are randomly initialized negative, the neuron will output 0 for all instances (since all features are positive for all instances). The output of the output neuron is therefore $\sigma(0 + b)$. With $b >= 0$, all instances are assigned to class 1. With $b < 0$, all instances are assigned to class 0.