I have data on ~800 observations that are relatively evenly fit between 5 categories. One of these categories is essentially a 'catch-all' group for observations that don't have any of the defining characteristics of the other 4 groups.

I'd like to train a classifier on this data that would ideally only design rules to predict membership of the 4 classes, and any that don't strongly fit into one of these groups is assigned to the fifth 'catch-all' group.

How could this best be achieved? If I include all the data and build a classifier on all 5 classes it will try to identify a defining pattern for the catch-all group, when really they are just the absence of a strong pattern.

I could exclude the catch-all observations and fit a classifier on the remaining data to predict membership of the 4 main groups, then any cases that don't meet a certain threshold could be assigned to the catch-all, but I'd prefer an automated approach.

  • 1
    $\begingroup$ My idea would be to perform a multinomial logistic regression for the four classes. Like this you get probabilities predicted. You can then set a threshhold. If for example the probability is above 0.8 you classify it into that class. If on all four regressions it never exceeds 0.8 you classify it as class 5. $\endgroup$
    – PeterD
    Jan 7, 2019 at 12:43
  • $\begingroup$ if it exceeds the threshhold multiple times you take the highest of course. You can also do that with other ML techniques. Set a threshhold, classify it as class 5 if the threshold is not exceeded or if it is exceeded take the class of the highest value. $\endgroup$
    – PeterD
    Jan 7, 2019 at 12:53
  • $\begingroup$ That's what I was thinking of, I was just wondering if there were any techniques available that had this functionality built-in. $\endgroup$ Jan 7, 2019 at 13:33
  • $\begingroup$ I dont know about that. $\endgroup$
    – PeterD
    Jan 7, 2019 at 14:20

1 Answer 1


The simple approach you can use, as said in comments, it to train a classifier on the data for the four classes, then set a threshold on the scores returned by the classifier, so that if neither of the scores for the classes passes the threshold, you classify the outcome as "other". The problem is however to choose the appropriate threshold. You may decide for some value like "probability >0.9", but if the classifier does not return calibrated probabilities (i.e. they don't promise to match the true probabilities), then you could see things like only-extreme scores, or only medium scores in your results. This means that your predefined threshold may be useless, and will be totally arbitrary.

By enforcing some threshold, you may inflate the false negatives rates for the four classes. So it would be the best if you had data about the "other" category and used it to find the threshold such that it maximizes some kind of metric of your choice for classifying to the five (four + "other") classes. However if you have such data, then it would be better just to train the classifier on the five classes instead.

Another possible problem with the threshold approach is that the "not class" may not be the "other" category that you are thinking of. Say that you trained a hot-dog vs not-a-hot-dog classifier on data containing photos of different kind of food and it learned to detect the sausage and based on this, to classify food as a hot-dog. Now, say that you used the classifier for photos of flowers. The classifier would be struggling to decide if it detected a sausage, or not, on each of the photos, but the decision that the photo is not a food may be completely irrelevant from detecting sausage-like shapes on the photos! This is another reason why it would be wise to have data on the "other" class.

Finally, you could also try two-step approach and first train an anomaly detection algorithm (see ) on the four class data and if it does not detect that the sample is anomalous, then use the four class classifier, otherwise detect the "other" class.

  • $\begingroup$ I see if I use data from the 'other' group to find the threshold then I may as well use this group's data in the classification in the first place, however, I have found that in doing so the classifier tries to find patterns in the data that identifies this 'other' group when really it is defined by absence of any the other 4 signals. In this particular application an increase in false negatives is more favourable than increasing FPR which could happen with including this data in a 5-group model or initial anomaly detection $\endgroup$ Jan 7, 2019 at 16:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.