How to choose a method for binary classifier based on only positive and unlabelled examples?

Question

I need to build a binary classifier with machine learning, as I fail to manually choose a combination of features to achieve minimal fraction of false positives.

What is best practice for choosing a ML method for building a binary classifier, specifically in Supervised Learning / Semi-Supervised PU (Positive/Unknown) group of methods? How to choose between Supervised Learning and Semi-Supervised/PU? What factors should I consider in making a choice of a specific method?

How can I practically check what kind of situation I have (like eg., as I saw in other questions, how to check whether the input set is separable; whether there's a strong correlation between features etc)?

If there are several methods which are equally good for my problem, which are easiest to start with, if I'm most comfortable with MacOS command line and Excel? (but don't mind traditional programming languages like Python)

I'm new to machine learning, so if that sound too broad, what would you recommend to read/watch to better understand my options?

---

To give some details of the problem I'm solving:

• I need to detect between "fake" and "genuine" objects
• I have a manually-created collection of objects that are known to be "fake", a subset of population that are "unknown" and I need to find "fakes" in
• I can easily create a set of "genuine" objects if that can seriously help; the problem here is 99% of them are quite "distant" from the "fakes" area.
  • However, if (with eg. SVM) I find a way to build a function to identify objects lying in the "grey" area, I would readily classify them by hand in order to use that for training a classifier

• choosing between precision and recall, I would prefer "minimal False Positives" while allowing large amount of "False Negatives"

• some numbers:

  • the collection of "fake" objects are several thousands
  • the "unknown" subset of population which is most fruitful to explore is 3x-5x times larger, but I can make even larger with less-relevant objects if required.
  • I have identified 20-30 numerical features that can be used in training
  • about 5-7 features are intuitively most "relevant" as they are most direct measures of suspicion
  • it is relatively easy to build more features if there's a systematic method of choosing which of them are most promising to add

---

So far I managed to manually build a "compound" function which sorts "unknown" objects by a combination of several "suspicion" metrics; the function is a linear combination of them.

There are about 1-2% of "genuine" objects that I manually identified in the "unknowns" when building that "compound" function.

When building the compound function, I tried to maximize the number of fake/unknown objects that occur before the first known "genuine" object. As the function is built of "the higher-the more-suspicious" metrics (features), what is "before" the known-genuines is higly likely fakes. However, I know that there are also many "fakes" in what is "after" each of genuines I'm aware of.

I'm willing to add more details if that can help to answer my question.

(For reference, my earlier question on datascience SE on the same problem)

Answer 1

I've worked on two projects like this, and can suggest two different approaches.

Approach one: You have a small number of examples labelled as "fake", and a lot of unlabelled examples. In the set of unlabelled examples, you expect most to be "genuine" and a few to be "fake".

In this case, I would train a classifier using downsampling, i.e. train lots of individual classifiers which use all/most of the known fake examples, and a subset of the unknown examples, but you give all of the unknown examples the label "genuine". You might get away with this, as the majority of them will be genuine, so each classifier will only see a small number of mislabelled examples.

In order to do actual detection, you iterate over the unknown example in chunks. E.g., (do this ten times), remove 10% of the data, train a classifier using the remaining 90% of the unknown examples (labelled as genuine) and all of the fake examples. Apply the predictions of this classifier to the 10% and see what fraction flag up as fake.

In order to get a gauge of how you're doing, you could also hold out some of the fake examples from the training set, and see how well your algorithm is doing at detecting known fakes.

Likewise, see what fraction of the 10% unknowns it is flagging as fake. Is it 1%, 5%, 50% ? Is the number plausible?

Approach 2: You think a sizeable fraction of the unlabelled data is fake. In this case, I'd suggest an unsupervised approach. You do some clustering, and see whether your labelled fake examples fall disproportionately into any of the clusters. If so, and if some unlabelled examples are in those same clusters, it's reasonable to speculate that there's an elevated probability that those unlabelled examples are fakes.

Note, I've assumed in all of this that you can't go out and get more labels. If it's possible to request more labels, there are semi-supervised/active learning approaches you could consider. I've less experience of these

Answer 2

So I will try an answer even if I have no direct experience with problems with a mixture of labeled and unlabeled data. Your problem seems similar to novelty detection, or anomaly detection, so you can search this site for the tags anomaly-detection and novelty-detection. One problem with searching for information is that this problems seem to go under multiple names, PU-learning, one-class learning, semi-supervised learning.

If not all examples are labeled, usual supervised learning seems difficult ... so here are some papers you could look at: Learning with Positive and Unlabeled Examples Using Weighted Logistic Regression,
Generative Probabilistic Models for Positive-Unlabeled Learning, This post is relevant: one class (positive and unlabeled) classification R package

There are also some R packages on CRAN, like AdaSampling, RSSL, ssc and SSL (and possibly others).

And maybe: start with some descriptive statistics/visualization of the labeled "fake" objects ... are they "fake" in some similar way, or all in its own way? Maybe something relevant in some of this: Posts on PU-learning or posts on one-class learning.