I need to build a binary classifier with machine learning, as I fail to manually choose a combination of features to achieve minimal fraction of false positives.
What is best practice for choosing a ML method for building a binary classifier, specifically in Supervised Learning / Semi-Supervised PU (Positive/Unknown) group of methods? How to choose between Supervised Learning and Semi-Supervised/PU? What factors should I consider in making a choice of a specific method?
How can I practically check what kind of situation I have (like eg., as I saw in other questions, how to check whether the input set is separable; whether there's a strong correlation between features etc)?
If there are several methods which are equally good for my problem, which are easiest to start with, if I'm most comfortable with MacOS command line and Excel? (but don't mind traditional programming languages like Python)
I'm new to machine learning, so if that sound too broad, what would you recommend to read/watch to better understand my options?
To give some details of the problem I'm solving:
- I need to detect between "fake" and "genuine" objects
- I have a manually-created collection of objects that are known to be "fake", a subset of population that are "unknown" and I need to find "fakes" in
- I can easily create a set of "genuine" objects if that can seriously help; the problem here is 99% of them are quite "distant" from the "fakes" area.
- However, if (with eg. SVM) I find a way to build a function to identify objects lying in the "grey" area, I would readily classify them by hand in order to use that for training a classifier
choosing between precision and recall, I would prefer "minimal False Positives" while allowing large amount of "False Negatives"
- the collection of "fake" objects are several thousands
- the "unknown" subset of population which is most fruitful to explore is 3x-5x times larger, but I can make even larger with less-relevant objects if required.
- I have identified 20-30 numerical features that can be used in training
- about 5-7 features are intuitively most "relevant" as they are most direct measures of suspicion
- it is relatively easy to build more features if there's a systematic method of choosing which of them are most promising to add
So far I managed to manually build a "compound" function which sorts "unknown" objects by a combination of several "suspicion" metrics; the function is a linear combination of them.
There are about 1-2% of "genuine" objects that I manually identified in the "unknowns" when building that "compound" function.
When building the compound function, I tried to maximize the number of fake/unknown objects that occur before the first known "genuine" object. As the function is built of "the higher-the more-suspicious" metrics (features), what is "before" the known-genuines is higly likely fakes. However, I know that there are also many "fakes" in what is "after" each of genuines I'm aware of.
I'm willing to add more details if that can help to answer my question.
(For reference, my earlier question on datascience SE on the same problem)