Suppose I train a multi-class classifier with K classes. However, in a practical application, an input can belong to none of the K classes. Thus, the K+1 class is introduced. The training data for this out-of-domain class is needed of course. The training of such classifier and usage is no different than the K-class classifier.
My question is: Why can't we get away with the original K binary classifiers including the case of out-of-domain input?
If an object belongs to one of of K classes, its score (or confidence) is 1.0 for the respective class. And it is zero for other K-1 classes. This is the case of highest certainty or lowest entropy.
If input belongs to none of the K classes, its score is 1/K for all K classes. This is the case of highest uncertainty or maximum entropy.
In a practical setting, I would compute entropy of the discrete distribution (scores/confidences must sum to one). If it is very close to uniform, the decision is "out-of-domain". If entropy is very low, then the classifier's prediction is the class for which the score is the highest (classical usage).
Is it possible to construct such classifier? If not, what are the reasons? A link to a publication would be great.
By extension, would that work for a multiple layer neural network? The last layer in the feed-forward neural network is essentially a linear classifier with a softmax at the end.