I am building an image classification network in tensorflow(several convolutional layers and fully connected layers, then softmax cross entropy, optimize using Adam with a learning rate of 1e-4).

Without dropout, I can get a pretty good performance. Though the loss is still very high, even the in-sample error approaches zero.

With dropout (dropout rate less than 0.25), the accuracy will gradually increase and loss will gradually decrease first. Then, accuracy will suddenly drop to 1/(# of class), and loss will also stay close to a small constant.

Does anyone know why that might happen? how I can solve this issue? Thank you


When you increase dropout beyond a certain threshold, it results in the model not being able to fit properly. Intuitively, a higher dropout rate would result in a higher variance to some of the layers, which also degrades training. Dropout is like all other forms of regularization in that it reduces model capacity. If you reduce the capacity too much, it is sure that you will get bad results.

The solution is to not use such high dropout. If you must, lowering the learning rate and using higher momentum may help.

Furthermore, be careful where you use dropout. It is usually ineffective in the convolutional layers, and very harmful to use right before the softmax layer.

  • $\begingroup$ Thanks for your reply. I didn't apply it to convolutional layers or output layer. I only applied it to hidden fully connected layer. And this happened even with really small dropout rate (~5%). Since the model can overfit the training data hard without dropout, I don't think it's any thing related to model capacity. $\endgroup$ – Erebuxy Jul 17 '17 at 16:08

I think I met the same problem with this dramatic-drop-in-accuracy issue. I think the problem with this issue might be related to the softmax function and the cross-entropy that you defined(loss function). Mine issue comes exactly from this cross-entropy function, and I used one-hot-format labels BTW. Taken that classic way of cross-entropy would cause nan or 0 gradient if "predict_y" is all zero or nan, so when the training iteration is big enough, all weights could suddenly become 0. This is exactly the reason why we can witness a sudden and dramatic drop in training accuracy. I solved this problem by using "tf.nn.softmax_cross_entropy_with_logits" instead, which can handle the extreme case safely.

The alternative loss function is defined as follow: tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=ture_y,logits=predict_y))

You can give it a shot


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.