4
$\begingroup$

I have a model that is trained on n classes and has more than 95% accuracy on the test set.

The model is going to receive a mixture of images that are either from one of the n classes or from unknown classes. My guess / expectation was that in the latter case, the distribution of values on the last layer (softmax) would be more or less uniform. Then to spot an alien image, it would be enough to threshold the prediction layer.

Unfortunately that is not the case, and the model seem to express a very high level of confidence every time. Even worse (using keras):

>>> max(model.predict(np.random.rand(1, 299, 299, 3))[0])
1.0

So basically the model is 100% sure that random noise is one of the class.

How to detect the images that do not belong to one of the classes that the model was trained on?

Note: everything has been normalized and in Keras predict returns probability, not class numbers.

Improve this question
$\endgroup$
7
  • $\begingroup$ It's hard to say without reproducible code. Did you normalize the training set? If yes, check that your random input has the same normalization. Otherwise, what does 95% accuracy mean with respect to 5 classes? Does that mean each class has a 95% True Positive rate? $\endgroup$
    – Alex R.
    Commented Jun 6, 2017 at 21:17
  • $\begingroup$ Yes, everything is normalized between 0 and 1. To answer your second question, it means that each class has 95% accuracy. $\endgroup$
    – meto
    Commented Jun 6, 2017 at 21:47
  • 1
    $\begingroup$ Are you sure that predict retuns probabilities and not a class prediction? Often python apis have a predict_proba method. $\endgroup$
    – Matthew Drury
    Commented Jun 7, 2017 at 0:52
  • $\begingroup$ Good point, but yes, predict return a vector with probabilities, not the class itlsef. in general you would have to do argmax to get the class. but my point is that the model was way too confident of the prediction $\endgroup$
    – meto
    Commented Jun 7, 2017 at 3:48
  • 3
    $\begingroup$ This is a well known problem in deep learning, see e.g. Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images $\endgroup$
    – sebp
    Commented Aug 31, 2018 at 8:00

2 Answers 2

Reset to default
1
$\begingroup$

You have not informed your model that there are other options, so in some ways it may be unsurprising that it guesses one. The softmax function in particular is designed to accentuate even slight differences in its inputs.

I would suggest adding a "none of the above" class to your model with appropriate training images of this class. Depending on what your application is, these "none of the above" images may be pure noise (as in your example), or actual images that are not from any of your target classes. Since the class of images not from any of your target classes will likely be quite large, I'd suggest picking images that are likely to appear in your application.

If this is insufficient, then you may want to look into estimating distributions to more explicitly find the images that fall far outside the distribution of images observed during training.

Improve this answer
$\endgroup$
0
$\begingroup$

Your interpretation of how the NN would classify a random input is only correct for feedforward networks. If there are convolution layers (which I assume there are since your data is image) the random input becomes correlated before it reaches to the classification layer. My suggestion is to use adversarial NN so that while the classification NN is trained for classification loss, discriminating NN would be trained to identify the anomalies.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.