How to represent no-detection in a confusion matrix

Question

So let's say we have a multi-class classification problem and we want to represent the outcomes as confusion matrix. All the examples I'm finding on the web asume that all the elements are detected. I understand the idea of the confusion matrix but I can't see how one can represent the non-detected elements on it.

Answer 1

You very well may want to abstain from classifying certain examples. It may be more costly to classify incorrectly than to state that you are uncertain and unwilling to classify, for example in a recommendation system.

Consider non-detection a new class that you never observe in the training data. You can build in a penalty for non-detection in your loss function, if you want to start at the very beginning. However, a simpler solution is just to set a probability cutoff beneath which you abstain from classifying--e.g. if the vector of estimated class membership probabilities has maximum less than 60%, don't classify. As long as the loss function is symmetric in the various classes, this corresponds to a implicit penalty on misclassification. You can even set a threshold for each class if failing to classify has higher/lower cost for certain classes.

Answer 2

I am not sure that something like the image below counts as a confusion matrix, but it makes sense to me.

The columns represent the true categories, either A or B (there could be a C or D or Z category, sure). The rows represent the predictions, either A or B in a classical confusion matrix, but also with an "I don't know" row.

Then you can see how many true As were classified as A, how many were classified as Bs, and how many were not classified as either because the system does not know. As has been discussed a few places [1][2], there can be more decisions than observed outcomes.