# Quality metric for classifier with decision rule allowing “none of the above”

Let's suppose that I have classification model for n classes ($$n>1$$). The classifier returns a probability distribution over a set of classes. But if classifier is not sure (i.e. there is no probability greater than given threshold) I would like the model to return answer "none of the above". In other words it is classifier with the following decision rule:

$$class(x) = \left\{\begin{matrix} \underset{i}{arg\, max} \,p_i & {if } & \exists_i\, p_i \geq \mathfrak{p} \\ \text{none-of-the-above} & {if } & \forall_i\, p_i < \mathfrak{p} \end{matrix}\right.$$

where $$\mathfrak{p}$$ is a probability threshold and $$p_i$$ is a probability that object $$x$$ belongs to class $$i$$.

And the question is: how to measure model accuracy?

One idea could be a calculation of any standard metric (for example $$F_1$$ score) and then multiplying it by percent of predicted classes:

$$quality = F_1 \cdot {{\text{size of test set}\,-\,\text{number of "none-of-the-above" cases}}\over{\text{size of test set}}}$$

Is that good idea? Or there are other approaches?