I build a multiclass classifier. I want the classifier to predict a few samples with little false positives, rather then many samples with lots of false positives.

Therefore I want to choose a probability threshold which leads to high precision. The image illustrates the choice, with a threshold of 0.8 I am at my desired precision.

enter image description here

I implemented my classifier in a way, that when a class is predicted with a low probability (< 0.8) the returned class is 'undefined'.

For example

True Class:                1    2   1   3    ...
Prediction:                1    2   1   3    ...
Probability:               0.8  0.7 0.5 0.8  ...
Prediction with Threshold: 1   -1  -1   3    ...

But now I a not sure how to calculate the metrics for this classifier. Should I just include a new undefined class or should I remove those samples which result in undefined. Both feels somehow wrong.

How would one calculate the confusion matrix for a classifier with results as shown above ?

I could so far not find any literature where multiclass predictions with undefined classes are covered.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.