# Is there something like a confusion matrix for a probabilistic score rather than categories?

Imagine we have pictures of three animals: dogs, cats, and horses. We train our image classifier and get a confusion matrix, noticing that the model tends to predict that dogs are horses.

But then we read Cross Validated and learn that threshold-based scoring rules like accuracy have serious flaws, so we want to look at the predicted probabilities, not just take the category with the highest probability.

Is there a way to adopt the confusion matrix to the probability outputs to notice that the model tends to believe that dogs are horses? The idea that comes to mind is to take the sum of the predicted probabilities (which works out to be the same as a confusion matrix, if we do this to predicted probabilities that we "round" to give a probability of $$1$$ of the category with the highest probability). Has this been explored in any literature?

I have never come across such a thing in literature, but it is a very interesting idea. Firstly, I'd like to point out that normalised confusion matrices exist (I know this isn't what you are asking for but it will illustrate a point I'm going to make, so just bare with me); for these types of confusion matrix there is some form of normalisation such that rows or columns sum to 1, the matrix has a norm of 1, or individual elements are normalised relative to the total number of samples. This means, of course, that a confusion matrix can contain entries which are in the range $$[0,1]$$ instead of the typical confusion matrix where entries are in range $$[0, NumSamples]$$, it encapsulates the same relationships as an un-normalized confusion matrix but simply with the values scaled.