Classification with a default class

Question

I have data on ~800 observations that are relatively evenly fit between 5 categories. One of these categories is essentially a 'catch-all' group for observations that don't have any of the defining characteristics of the other 4 groups.

I'd like to train a classifier on this data that would ideally only design rules to predict membership of the 4 classes, and any that don't strongly fit into one of these groups is assigned to the fifth 'catch-all' group.

How could this best be achieved? If I include all the data and build a classifier on all 5 classes it will try to identify a defining pattern for the catch-all group, when really they are just the absence of a strong pattern.

I could exclude the catch-all observations and fit a classifier on the remaining data to predict membership of the 4 main groups, then any cases that don't meet a certain threshold could be assigned to the catch-all, but I'd prefer an automated approach.

Answer 1

The simple approach you can use, as said in comments, it to train a classifier on the data for the four classes, then set a threshold on the scores returned by the classifier, so that if neither of the scores for the classes passes the threshold, you classify the outcome as "other". The problem is however to choose the appropriate threshold. You may decide for some value like "probability >0.9", but if the classifier does not return calibrated probabilities (i.e. they don't promise to match the true probabilities), then you could see things like only-extreme scores, or only medium scores in your results. This means that your predefined threshold may be useless, and will be totally arbitrary.

By enforcing some threshold, you may inflate the false negatives rates for the four classes. So it would be the best if you had data about the "other" category and used it to find the threshold such that it maximizes some kind of metric of your choice for classifying to the five (four + "other") classes. However if you have such data, then it would be better just to train the classifier on the five classes instead.

Another possible problem with the threshold approach is that the "not class" may not be the "other" category that you are thinking of. Say that you trained a hot-dog vs not-a-hot-dog classifier on data containing photos of different kind of food and it learned to detect the sausage and based on this, to classify food as a hot-dog. Now, say that you used the classifier for photos of flowers. The classifier would be struggling to decide if it detected a sausage, or not, on each of the photos, but the decision that the photo is not a food may be completely irrelevant from detecting sausage-like shapes on the photos! This is another reason why it would be wise to have data on the "other" class.

Finally, you could also try two-step approach and first train an anomaly detection algorithm (see anomaly-detection) on the four class data and if it does not detect that the sample is anomalous, then use the four class classifier, otherwise detect the "other" class.