Deep learning high dropout causes high model confidence scores I am training an NLP classifier that maps input sentences to 1 of 50 categories.  
The model is a CNN language model, in which each input example is a 2d tensor of sentence length by word embedding vector size.
Every example in the test set is not present in training.  I achieve the best test score on the data when regularizing with very high dropout (> 0.9).  I can train this model so that test approx = train score after training.
However, the predictions from this model are all very high confidence.  Furthermore, there are many predictions with confidence = 1.0.  How can this be possible on unseen data?

If I use L2 regularization instead the problem goes away, however, the model cannot reach the same test score.

Could there be any valid theoretical reason why this might occur?  Any ideas to solve the problem? 
It could be that this is just a very, very good model.  The data contains many annotation errors, which explains the high confidence incorrect predictions.  There are low confidence predictions, just not as many as I would expect to see...  But the confidence = 1 on any test data seems wrong to me.
 A: Firstly, be aware that the outputs of a normal neural network have nothing to do with the confidence of the predictions. If the output of the softmax layer is nearly 1 for some class, that does not automatically mean that the network is almost certain of the prediction. Softmax only normalizes outputs so they sum up to 1, but it does not provide any confidence about that prediction.
That said, I think you simply did a good job training the model for the task. Since you have not given any details about the dataset, I guess it might simply be that the task is relatively easy. A well regularized model should perform comparably good on training and testing data, and 0.9 dropout rate is very strong regularization.
If you are suspicious, try inspecting the data. Make sure the vector representation of validation set sentences is not the same as the sentences in the training data.
A: To elaborate on this problem.  This is know as a poorly calibrated model, where the network outputs cannot be interpreted as predicted probabilities.  You can visualize this by plotting accuracy per confidence bin for the test set.  A well calibrated model will resemble the left hand plot, whereas a badly calibrated model will resemble the right plot.

Indeed when plotting this for my model above, it was very poorly calibrated.
This paper investigates causes and techniques to solve the problem:
https://arxiv.org/pdf/1706.04599.pdf
