Suppose I train a multi-class classifier with K classes. However, in a practical application, an input can belong to none of the K classes. Thus, the K+1 class is introduced. The training data for this out-of-domain class is needed of course. The training of such classifier and usage is no different than the K-class classifier.

My question is: Why can't we get away with the original K binary classifiers including the case of out-of-domain input?

If an object belongs to one of of K classes, its score (or confidence) is 1.0 for the respective class. And it is zero for other K-1 classes. This is the case of highest certainty or lowest entropy.

If input belongs to none of the K classes, its score is 1/K for all K classes. This is the case of highest uncertainty or maximum entropy.

In a practical setting, I would compute entropy of the discrete distribution (scores/confidences must sum to one). If it is very close to uniform, the decision is "out-of-domain". If entropy is very low, then the classifier's prediction is the class for which the score is the highest (classical usage).

Is it possible to construct such classifier? If not, what are the reasons? A link to a publication would be great.

By extension, would that work for a multiple layer neural network? The last layer in the feed-forward neural network is essentially a linear classifier with a softmax at the end.


It is worth pointing out that your problem of training a classifier to detect $k$ known classes and 1 "unknown" class can be split into a two-step problem. We would like a model to 1) classify a point as being "known" or "unknown" and 2) if "known", classify the point into one of $k$ classes.

Since you are apparently familiar with methods to classify things into $k$ classes, your problem has been reduced to classifying points as being "known" or "unknown". This is what is commonly called novelty detection or anomaly detection. For continuous input, a popular method is one-class SVM, which, in some sense, tries to find a "soft" convex hull for the training data. Other methods for anomaly detection abound in time-series and for other types of input.

  • $\begingroup$ "If input belongs to none of the K classes, its score is 1/K for all K classes." This almost never happens for real classifiers. In my experience neural networks are somehow 90% confident even when you feed them total rubbish. $\endgroup$ – Scott Sep 4 at 3:30
  • $\begingroup$ The score of 1/K for unknown input is used during training. At the prediction stage, the distribution over K classes in practice will of course almost never be exactly uniform. I am wondering whether such classifier can be trained. If not, what are the clear reasons why it is not even worth setting up and running this experiment? $\endgroup$ – Vladislavs Dovgalecs Sep 4 at 17:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.