I have successfully trained a ResNet50V2 model. I used transfer learning on ImageNet database, and launch the inference on the unseen data. I obtain a high accuracy (>80%), but when I print the probability vector for each label, I obtain strangely high values. The sum of the elements is one, sure, but it is like the CNN is almost sure when it performs a prediction. For the predicted label I recover a probability of 1 and this trend is quite common in all the 200 predictions. In the last dense layer I use the softmax activation function. Is there an error?
6
-
1$\begingroup$ Might be overfitting, nearly interpolating? What's the score on the training set? $\endgroup$– Ben ReinigerApr 20 at 21:59
-
1$\begingroup$ I didn't mean information leak, nor that the predictions might not be nearly-0s-and-1s on any of the different sets. I meant, if the training score is nearly perfect, then the model is correctly calibrating: it has learned exactly which inputs lead to which outputs on the training set. It will continue being similarly confident on the test set, but now will be wrong more often (hence the drop in accuracy). $\endgroup$– Ben ReinigerApr 21 at 13:46
-
1$\begingroup$ The accuracy is 93% on the train set, 92% in the validation set, and 89 on the test set. $\endgroup$– Jonny_92Apr 21 at 14:13
-
1$\begingroup$ Not as dramatic as I had expected under my theory. Maybe still: among the misclassified in the training set, are the probabilities similarly all-or-nothing? $\endgroup$– Ben ReinigerApr 21 at 14:17
-
1$\begingroup$ Exactly, this is the pattern. When I evaluate on the training set also the wrong predictions manifest an all-or-nothing behavior of the probability. $\endgroup$– Jonny_92May 2 at 6:21
|
Show 1 more comment