2
$\begingroup$

This is a homework assignment, so I have strict requirements. In that assignment, I am required to train a neural network with sigmoid activation function to achieve 0 training error rate [I know, wrong thing to do, overfit and such, but I am required to do so :( ].

I have implemented the gradient descent algorithm and things seems fine, but it never reaches 0% training error rate. I diagnosed it a bit and realize all the hidden layer neuron are dead (their output is very close to 0 or 1), so there is no way I can update their weights. The output layer just can't solve the non-linear problem even if their weights can be adjusted.

My questions are:

(1) Is this a well known problem? (2) How do I deal with it?

I saw some post talking about using a different activation functions, unfortunately that isn't an option for me for this assignment.

Improve this question
$\endgroup$
1
  • $\begingroup$ Try initializing the weights with smaller values. $\endgroup$
    – Aaron
    Commented Mar 5, 2017 at 0:35

1 Answer 1

Reset to default
1
$\begingroup$

As you said, a different activation function wold be better, but you can't. I would suggest you to try different initializations of the weights: those can affect the final result. You can try a different optizimization technique, too, maybe adding momentum could help you.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.