I am training an NLP classifier that maps input sentences to 1 of 50 categories.
The model is a CNN language model, in which each input example is a 2d tensor of sentence length by word embedding vector size.
Every example in the test set is not present in training. I achieve the best test score on the data when regularizing with very high dropout (> 0.9). I can train this model so that test approx = train score after training.
However, the predictions from this model are all very high confidence. Furthermore, there are many predictions with confidence = 1.0. How can this be possible on unseen data?
If I use L2 regularization instead the problem goes away, however, the model cannot reach the same test score.
Could there be any valid theoretical reason why this might occur? Any ideas to solve the problem?
It could be that this is just a very, very good model. The data contains many annotation errors, which explains the high confidence incorrect predictions. There are low confidence predictions, just not as many as I would expect to see... But the confidence = 1 on any test data seems wrong to me.