Accuracy measure for model which sometimes produces no prediction

Question

I have a model which produces no output for some inputs. What's a reasonable way to measure the performance of the model against a data set, taking the "missing output" into consideration?

And is there a sane way to compare this performance to a model with a continuous output?

For example...

The model I have is a classifier of the form:

IF input between 0 and 10 THEN true
ELSE if input between 11 and 20 THEN false

This model will produce no classification for inputs below 0 or above 20. This is by design: the induction process has decided this range of inputs is the predictable part of the problem.

To compute something like accuracy, I could treat "no output" as just being "wrong" (neither a true positive or true negative). But this doesn't seem right: I feel like I'm missing a bunch of ways to approach this situation.

Compare this to a logistic regression model, where I have the probability of some classification, and presumably a threshold for true/false classifications. In this case, ROC AUC might be commonly used. Is there any sane way to compare a logistic regression model to one which produces no output for some inputs?

Pointers much appreciated: I don't even have the right language to describe this situation.

Answer 1

You essentially have a multiclass classification problem. If I understand correctly, you have two "real" classes, but your classifier might output not only one of these, but also a third "none of the above" class.

It actually does not matter whether this third "class" ever appears in your actual sample.

I would propose that you use a probabilistic classifier, which does not output hard 0-1 classifications, but class membership probabilities, which in extreme cases could be up to 100% for one class, and 0% for the others.

Then evaluate these predictions using proper scoring-rules, like the log score or the Brier score. (If you actually have 0% predictions for a class that does turn up, the log score will try to take $\log 0$ and throw an error, so the Brier score might be better.)

One consequence would be that a classifier that at least puts a little probability on the class that does turn up rather than being 100% certain for "none of the above" will be preferred (other things being equal). I assume this is what you want.

More info can be found at Evaluating multiclass imbalanced problem per class. You might also be interested in Why is accuracy not the best measure for assessing classification models? and Reduce Classification Probability Threshold. (Apologies for the shameless self-promotion. I just tend to be most familiar with threads I participated in.)