I build a multiclass classifier. I want the classifier to predict a few samples with little false positives, rather then many samples with lots of false positives.
Therefore I want to choose a probability threshold which leads to high precision. The image illustrates the choice, with a threshold of 0.8 I am at my desired precision.
I implemented my classifier in a way, that when a class is predicted with a low probability (< 0.8) the returned class is 'undefined'.
True Class: 1 2 1 3 ... Prediction: 1 2 1 3 ... Probability: 0.8 0.7 0.5 0.8 ... Prediction with Threshold: 1 -1 -1 3 ...
But now I a not sure how to calculate the metrics for this classifier. Should I just include a new undefined class or should I remove those samples which result in undefined. Both feels somehow wrong.
How would one calculate the confusion matrix for a classifier with results as shown above ?
I could so far not find any literature where multiclass predictions with undefined classes are covered.