Why accuracy gradually increase then suddenly drop with dropout

Question

I am building an image classification network in tensorflow(several convolutional layers and fully connected layers, then softmax cross entropy, optimize using Adam with a learning rate of 1e-4).

Without dropout, I can get a pretty good performance. Though the loss is still very high, even the in-sample error approaches zero.

With dropout (dropout rate less than 0.25), the accuracy will gradually increase and loss will gradually decrease first. Then, accuracy will suddenly drop to 1/(# of class), and loss will also stay close to a small constant.

Does anyone know why that might happen? how I can solve this issue? Thank you

Answer 1

When you increase dropout beyond a certain threshold, it results in the model not being able to fit properly. Intuitively, a higher dropout rate would result in a higher variance to some of the layers, which also degrades training. Dropout is like all other forms of regularization in that it reduces model capacity. If you reduce the capacity too much, it is sure that you will get bad results.

The solution is to not use such high dropout. If you must, lowering the learning rate and using higher momentum may help.

Furthermore, be careful where you use dropout. It is usually ineffective in the convolutional layers, and very harmful to use right before the softmax layer.

Answer 2

I think I met the same problem with this dramatic-drop-in-accuracy issue. I think the problem with this issue might be related to the softmax function and the cross-entropy that you defined(loss function). Mine issue comes exactly from this cross-entropy function, and I used one-hot-format labels BTW. Taken that classic way of cross-entropy would cause nan or 0 gradient if "predict_y" is all zero or nan, so when the training iteration is big enough, all weights could suddenly become 0. This is exactly the reason why we can witness a sudden and dramatic drop in training accuracy. I solved this problem by using "tf.nn.softmax_cross_entropy_with_logits" instead, which can handle the extreme case safely.

The alternative loss function is defined as follow:

tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=ture_y,logits=predict_y))

You can give it a shot.