I have a CNN implemented on TensorFlow. I am using it on the MNIST dataset.

9/10 times, when I start training it, it very quickly gets up past 90%.

However, once in a while, it zeros in on 9% performance. Is this a bug in my code, or a CNN phenomenon? (I remember reading something about encountering singularities in poorly initialized neural nets--I am just doing a truncated normal matrix for my weights)

Hopefully, it is a CNN thing associated with the gradient descent and not my code...because this intermittent success and failure is very difficult to debug...


Although this doesn't guarantee your code is 100% correct (hehe), yes, this can be due to a neural network phenomenon. If your initialization is poor (or even if it was good) and your learning rate is too high, then you might skip the optimal point you are looking for, and because of the high learning rate, you might never turn back to it.

this figure (from here) summarizes it very nicely: (the red circle at the bottom left is your starting point, the x axis is your parameter space and the y axis is the error.)

enter image description here

  • $\begingroup$ I have the same problem with a model that I have been working with. I want to ask whether is this the only possible reason for this problem..? Or is there any other reasons behind it..? $\endgroup$ – Ramesh-X Oct 4 '17 at 16:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.