How to decide if a test data point doesn't belong to any of the classes in the train data?

Question

I am working on a multi-class classification problem. I was wondering if there is way to decide if a test sample could not belong to any of the classes in the training set? Simply put I want to classify if the data point belongs to an unseen class - 'others'

For now, I am thinking of extracting probabilities of how much each data point belongs to each class. If these probabilities are skewed (i.e. are weighed highly for some classes and very low for majority of classes), then it belongs to one of the train set classes. If these probabilities are uniform (say almost equally distributed) then it belongs to an others class. But obviously this method might give bad results and I would also need to empirically set the thresholds for uniformity and skewness.

Answer 1

This sounds like Outlier Detection to me. You should consider Unsupervised (Hierarchical Clustering) and Semi-Supervised methods.

For example, you could use One-class classification. Label all of your training as 'Class1' and see if the classifier assigns it to 'Class1' or 'Other'.

Using Python and data with some kind of distance function, have a look at:

• http://scikit-learn.org/stable/modules/generated/sklearn.svm.OneClassSVM.html
• http://scikit-learn.org/stable/auto_examples/svm/plot_oneclass.html

If you want to use the technique you mentioned, here is a slightly different approach to get better results: Create One-vs-All classifiers. For each binary classifier, the class '1' is fitted against all the other classes '0'. Instances that never made it to class '1' for all classifiers can then be labeled as 'Other'.

http://scikit-learn.org/stable/modules/generated/sklearn.svm.OneClassSVM.html

Yes, I like SVMs.