Is it possible to alter binary classification models to do postive-unlabeled learning in Pyspark?

Question

I'm learning how to use pyspark, and I'm wondering if it has any ways to implement positive-unlabeled learning? From searching this question I haven't been able to find any examples specific in spark for python (only java which I am not familar with).

I'm looking to do positive-unlabeled machine learning that has the potential to scale, so whilst I can get PU-learning running in packages focused on scikit-learn models for this I want to know if it would be possible to do in PySpark.

Currently not sure where to start/look to answer this question. I've been looking in the spark docs (https://spark.apache.org/docs/latest/api/python/reference/pyspark.ml.html#classification) and I see they offer models that can do binary classification. I'm still learning about machine learning, so I'm wondering if it would be possible for me to use a binary classifier but somehow re-weigh the negative class so it's more like it's unlabelled vs positive?

Answer 1

It depends on the PU-learning method. PU often comes down to different sampling concepts and/or relabelling your data - but the algorithms are usually the same as in binary classification. For example some pu algorithms use an estimated fraction of positives and negatives in your unlabaled data to use some of it for training as well. So in this case, you can simply consider implementing some kind of wrapper for binary classification packages in PySpark to include this.

To be more specific, you must provide additional information on problem statement, data and pu-method you want to use.

Also, the term pu-learning is not that popular to begin with. Maybe you can find something interesting if you look for relevant semi-supervised packages/libs instead.