# Multiclass classifier with undefined prediction, how to calculate metrices

I build a multiclass classifier. I want the classifier to predict a few samples with little false positives, rather then many samples with lots of false positives.

Therefore I want to choose a probability threshold which leads to high precision. The image illustrates the choice, with a threshold of 0.8 I am at my desired precision.

I implemented my classifier in a way, that when a class is predicted with a low probability (< 0.8) the returned class is 'undefined'.

For example

True Class:                1    2   1   3    ...
Prediction:                1    2   1   3    ...
Probability:               0.8  0.7 0.5 0.8  ...
Prediction with Threshold: 1   -1  -1   3    ...


But now I a not sure how to calculate the metrics for this classifier. Should I just include a new undefined class or should I remove those samples which result in undefined. Both feels somehow wrong.

How would one calculate the confusion matrix for a classifier with results as shown above ?

I could so far not find any literature where multiclass predictions with undefined classes are covered.