I am building an image classification network in tensorflow(several convolutional layers and fully connected layers, then softmax cross entropy, optimize using Adam with a learning rate of 1e-4).

Without dropout, I can get a pretty good performance. Though the loss is still very high, even the in-sample error approaches zero.

With dropout (dropout rate less than 0.25), the accuracy will gradually increase and loss will gradually decrease first. Then, accuracy will suddenly drop to 1/(# of class), and loss will also stay close to a small constant.

Does anyone know why that might happen? how I can solve this issue? Thank you