Below was tried in R, but any general solution would be highly appreciated:
I have 2 class samples (both classes are balanced). I want to create a classifier, where I only care about 1 class (So, if the classifier tells me it belongs to class "2", then I don't care).
HOWEVER, when the classifier tells me it is class "1", then it must be as accurate as possible. So I am looking for highest rate of TP vs FP. When I use ALL my data for training (and testing) I get poor results.
Here is what I am willing to do in order to achieve this: Let's say I have 5,000 samples. I have no problem that TP will not include all (so the classifier can be setup to classify many samples as FN or TN). BUT, the FP must be as low as possible.
Anybody has an idea how to do this ? When using a tree, for example, it tries to find the BEST for both classes. I need kind of pre filter or another sort of way to take 2 class, and use as "one sided".
I tried to play with cost matrix and so on, nothing really helped.
This, by the way, sounds to me,. like a true common real-life problem, yet I have no found any written solutions on this
IN OTHER WORDS: I am trying to find the largest subset of the entire data, which gives the best TP/FP rate. I have no idea how to find this subset. I did brute force trials, where each time I took one single feature (I have 100), did a subset, based on different threshold of this feature, and then build a tree on the subset. This, of course, is not a real smart idea
REAL LIFE EXAMPLE: I have a patient arriving for a series of tests, in order to find out if he has a "X" disease. I perform different tests, and for each one get a results. I need to be at least 80% accurate weather the patient has the "X" disease or not. If I use all tests, then I will not achieve 90% accuracy. However, I am willing to tell to different patients that I cannot predict, however, I cannot be more than 10% FP.
So, out of 100 patients, I am willing to tell 30% of them that I do not know, however, for the other 70%, I want to be at least 90% accurate to say they have the "X" disease.
So, I need to find the subset of patients, which for them, the classifier will be 90% TP and 10% FP
The literature is full of feature selection and feature reduction. I haven't seen anything on data selection (which is what I am trying to do here. I am willing to "give up" on patients, for the purpose that on those which I classify, I need to be very good)