This is a homework assignment, so I have strict requirements. In that assignment, I am required to train a neural network with sigmoid activation function to achieve 0 training error rate [I know, wrong thing to do, overfit and such, but I am required to do so :( ].
I have implemented the gradient descent algorithm and things seems fine, but it never reaches 0% training error rate. I diagnosed it a bit and realize all the hidden layer neuron are dead (their output is very close to 0 or 1), so there is no way I can update their weights. The output layer just can't solve the non-linear problem even if their weights can be adjusted.
My questions are:
(1) Is this a well known problem? (2) How do I deal with it?
I saw some post talking about using a different activation functions, unfortunately that isn't an option for me for this assignment.