Below was tried in R, but any general solution would be highly appreciated:

I have 2 class samples (both classes are balanced). I want to create a classifier, where I only care about 1 class (So, if the classifier tells me it belongs to class "2", then I don't care).

HOWEVER, when the classifier tells me it is class "1", then it must be as accurate as possible. So I am looking for highest rate of TP vs FP. When I use ALL my data for training (and testing) I get poor results.

Here is what I am willing to do in order to achieve this: Let's say I have 5,000 samples. I have no problem that TP will not include all (so the classifier can be setup to classify many samples as FN or TN). BUT, the FP must be as low as possible.

Anybody has an idea how to do this ? When using a tree, for example, it tries to find the BEST for both classes. I need kind of pre filter or another sort of way to take 2 class, and use as "one sided".

I tried to play with cost matrix and so on, nothing really helped.

This, by the way, sounds to me,. like a true common real-life problem, yet I have no found any written solutions on this

IN OTHER WORDS: I am trying to find the largest subset of the entire data, which gives the best TP/FP rate. I have no idea how to find this subset. I did brute force trials, where each time I took one single feature (I have 100), did a subset, based on different threshold of this feature, and then build a tree on the subset. This, of course, is not a real smart idea

REAL LIFE EXAMPLE: I have a patient arriving for a series of tests, in order to find out if he has a "X" disease. I perform different tests, and for each one get a results. I need to be at least 80% accurate weather the patient has the "X" disease or not. If I use all tests, then I will not achieve 90% accuracy. However, I am willing to tell to different patients that I cannot predict, however, I cannot be more than 10% FP.

So, out of 100 patients, I am willing to tell 30% of them that I do not know, however, for the other 70%, I want to be at least 90% accurate to say they have the "X" disease.

So, I need to find the subset of patients, which for them, the classifier will be 90% TP and 10% FP

The literature is full of feature selection and feature reduction. I haven't seen anything on data selection (which is what I am trying to do here. I am willing to "give up" on patients, for the purpose that on those which I classify, I need to be very good)

  • $\begingroup$ If you are using classification trees in R, you can use the weights parameter in the tree function to give the desired class higher weighting. $\endgroup$
    – shane
    Jul 22, 2015 at 15:11

1 Answer 1


You may want to look into "classification with reject option" literature. Essentially, you want not only to output a predicted class, but also confidence of that prediction. They you can choose a confidence level, and select only those patients for which the confidence is high.

A simple approach to obtain confidence estimates would be to use a probabilistic classifier, say Naive Bayes. Then if p(class1|X) is close to 1, the confidence is very high. If p(class1|X) is close to 0.5 (or other threshold, depending on the cost matrix), then the confidence is low. If p(class1|X) is close to 0, then the confidence is high again, but this time it is for class 2.

  • $\begingroup$ I am using R. I have used the rpart and predict with the "prob". However, most results fall close to each other. If I set a threshold of 0.6, then I get about 1/6 of total samples, and yet, the prediction is not good. If I set to 0.7, then I really get a low amount of samples, and still my total accuracy is 0.52, very poor $\endgroup$ Jul 22, 2015 at 13:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.