What is it called when, in a classification task, it is acceptable that some data-points do not receive a label? And what classifiers are suitable?

I have a dataset with a two valued target variable. After manually tweaking the results of the CN2 rule-induction algoritm i get:

  • A few short rules that select sizable parts of the data (ie do not overfit)
  • These rules are meaningful to a domain expert

These rules accurately label data-points belonging in each group, but about 35% of the data-points are not labelled at all. The distribution of classes in the unlabelled data is the same as in the orginal dataset. In other words, the probability of falling in either group, for the unlabeled data-points, is the same as the a-priori probability.

These are acceptable results, because the number of false positives is low for both labels. In real life this kind of classification is common, think of a doctor who runs tests to exclude certain conditions.
It also is the best classification i have been able to come up with for this data-set.

My questions are:

  • Is there a name for this kind of classification?
  • What classifiers can do this out of the box? Here i manually tweaked rules generated bij CN2.

Any classification which returns probabilities (or at least some numeric score), can transformed in a classification with unknown. Suppose you have a binary classifier which returns the probability of an instance to belong to one of the classes. A probability closer to $1$ means a high probability to belong to class $A$, and a probability value closer to $0$ means a high probability to belong to class $B$ (to not belong to class $A$). You cand design your decision function such as that if the probability is belong a low threshold value (let's say 0.40) will be classified as $B$, if is greater than a high threshold value (let's say 0.60) will be classifier as $A$, and is undecided if the value is in between the threshold values.

Obviously, you can extend this idea to scores instead of probabilities, and also to multi-class algorithms. For the later case you can use a min threshold value for a discriminant function.

  • $\begingroup$ I am going to accept this answer because it gives a more generalized account of the method described in my question. However i was hoping to get answers referring to publications describing similar approaches. $\endgroup$
    – Ivana
    Jan 21 '15 at 15:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.