0
$\begingroup$

Please comment on the following method:

Model was trained on data labeled with 4 classes: header, question, answer an other. Predictions were done on a separate test set, nothing new here.

Then samples of every class with predicted probability lower then some threshold (for example 0.6) were assigned to a new class called 'Noclass', that model was not trained on, simply because dataset has no samples labeled 'Noclass'.

As a result the following Confusion Matrix was created:

enter image description here

In this matrix 'Noclass' has neither True Positives, nor False Positives, they are all zeroes. To my mind this is not a Confusion Matrix, but something else. I think, that artificially adding class to Confusion Matrix of a model, that was not trained on such class doesn't make sense at all.

And in general, is it possible to have a Confusion Matrix where TP and FP of some class are zero?

What do you think?

$\endgroup$
7
  • $\begingroup$ Should your confusion monotonic be square? No matter how you classify an observation, it is something, right? $\endgroup$
    – Dave
    Commented Feb 9, 2023 at 13:05
  • $\begingroup$ @Dave Please clarify what you mean. Thanks $\endgroup$
    – dokondr
    Commented Feb 9, 2023 at 20:59
  • $\begingroup$ Along one axis, you have the four true categories. Along the other axis, you have the five possible predicted categories (including an "no class" category). That would be a $4\times5$ (or $5\times4$) array, not a square. $\endgroup$
    – Dave
    Commented Feb 9, 2023 at 21:05
  • $\begingroup$ @Dave Sure it is not a square, does it mean that is not a Confusion Matrix? I think what is more important, is that the model that never predicts one of the classes is a bad model, does not make sense. $\endgroup$
    – dokondr
    Commented Feb 9, 2023 at 21:11
  • 1
    $\begingroup$ It isn't the classical "confusion matrix" unless it is square, but it still could be a useful visualization of what the predictions are compared to what they should be. // I disagree about a model being bad if it gives a "no class" classification. Yes, we want our models to predict the correct category every time, but that might not be realistic, and a model coming back and telling us, "I don't know, collect more data," could be quite useful and an honest reflection of the uncertainty. $\endgroup$
    – Dave
    Commented Feb 9, 2023 at 21:16

1 Answer 1

0
$\begingroup$

To answer your question "is it possible to have a Confusion Matrix where TP and FP of some class are zero?". Yes this is possible - it simply means the model has not predicted anything to be in that class.

A simple example would be if I threw a coin twice. Before throwing the coin, I predict it will land tails each time. The coin then lands on Heads and then Tails on the attempts.

My confusion matrix would then be:

Predicted H Predicted T
Actual H 0 1
Actual T 0 1

Assuming that Heads is "positive", then I will have no True Positives and no False Positives.

$\endgroup$
1
  • $\begingroup$ Yes, but the model that never predicts one of the classes is a bad model, does not make sense. $\endgroup$
    – dokondr
    Commented Feb 9, 2023 at 20:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.