# Confusion Matrix with a class created from lower probabilities of other classes

Please comment on the following method:

Model was trained on data labeled with 4 classes: header, question, answer an other. Predictions were done on a separate test set, nothing new here.

Then samples of every class with predicted probability lower then some threshold (for example 0.6) were assigned to a new class called 'Noclass', that model was not trained on, simply because dataset has no samples labeled 'Noclass'.

As a result the following Confusion Matrix was created:

In this matrix 'Noclass' has neither True Positives, nor False Positives, they are all zeroes. To my mind this is not a Confusion Matrix, but something else. I think, that artificially adding class to Confusion Matrix of a model, that was not trained on such class doesn't make sense at all.

And in general, is it possible to have a Confusion Matrix where TP and FP of some class are zero?

What do you think?

• Should your confusion monotonic be square? No matter how you classify an observation, it is something, right?
– Dave
Commented Feb 9, 2023 at 13:05
• @Dave Please clarify what you mean. Thanks Commented Feb 9, 2023 at 20:59
• Along one axis, you have the four true categories. Along the other axis, you have the five possible predicted categories (including an "no class" category). That would be a $4\times5$ (or $5\times4$) array, not a square.
– Dave
Commented Feb 9, 2023 at 21:05
• @Dave Sure it is not a square, does it mean that is not a Confusion Matrix? I think what is more important, is that the model that never predicts one of the classes is a bad model, does not make sense. Commented Feb 9, 2023 at 21:11
• It isn't the classical "confusion matrix" unless it is square, but it still could be a useful visualization of what the predictions are compared to what they should be. // I disagree about a model being bad if it gives a "no class" classification. Yes, we want our models to predict the correct category every time, but that might not be realistic, and a model coming back and telling us, "I don't know, collect more data," could be quite useful and an honest reflection of the uncertainty.
– Dave
Commented Feb 9, 2023 at 21:16

To answer your question "is it possible to have a Confusion Matrix where TP and FP of some class are zero?". Yes this is possible - it simply means the model has not predicted anything to be in that class.

A simple example would be if I threw a coin twice. Before throwing the coin, I predict it will land tails each time. The coin then lands on Heads and then Tails on the attempts.

My confusion matrix would then be:

Predicted H Predicted T
Actual H 0 1
Actual T 0 1

Assuming that Heads is "positive", then I will have no True Positives and no False Positives.

• Yes, but the model that never predicts one of the classes is a bad model, does not make sense. Commented Feb 9, 2023 at 20:56