As I understood the concept of semi-supervised learning is to train a classifier on the minimal available subset of correctly labeled data in order to predict the labels of a greater previously unlabelled dataset, therefore creating pseudo-labels.

Lastly, a supervised model is trained on this dataset containing labels and pseudo labels.

But what if I just have two labels say $\{0,1\}$ and $1$ is always observable but $0$ cannot be observed with certainty.

An example: suppose I have to train a model that gives me the probability of converting a certain firm into a client. I can observe my current clients (therefore label=$1$) but for nonclients, they can either be prospects, therefore, $1$ or not clients at all therefore true $0$.


This seems like a a case of PU learning — learning from positive-only and unlabeled examples. The original paper on this is here. One of the main results is the following:

Suppose that $y$ is the true class label (0 or 1), $s$ is the observed class label (0 if negative or unlabeled, 1 if positive AND labeled), and $x$ is our data. Then $p(y = 1 | x) = p(s = 1 | x) * c$, where $c$ is a constant. In fact, you can show that $c = p(s=1 | y=1)$.

So, how do we apply this in the real world? Well, based on our dataset, we can make the following statements for all $x$:

  1. $p(s = 1 | y = 1; x) = \alpha$ (assume this is a constant)
  2. $p(s = 0 | y = 1; x) = 1 - \alpha$
  3. $p(s = 1 | y = 0; x) = 0$ (all labeled examples must be positive)
  4. $p(s = 0 | y = 0; x) = 1$ (you don't ever label a negative example)

Through some probabilistic manipulation, we can show that $\alpha = \frac{1}{c}$. So our problem is really about estimating $\alpha$. Suppose you had a magic function $h(x) = p(y=1 | x)$; i.e. you have the classifier that you want to learn. If you were to prove this in real life, one way to do this is to show that $\mathbb{E}[h(x) | y=1] = \alpha$. From this, we have the following algorithm:

$\alpha \sim= \frac{1}{|V^+|} \sum_{x \in |V^+|} h(x)$.

This means that you can naively train a model to discriminate between labeled positive examples and everything else, then rescale via $\alpha$ to obtain a decent estimator. That is; you don't have magical function $h(x)$, but the naive model behaves the same way on positive labeled examples as the ideal magical model.

Concretely, in a logistic regression toy example with $x \in \mathbb{R}^2$, if you draw the decision boundaries obtained from a naive vs. rescaled classifier, you will notice that this rescaling "shifts" the decision boundary linearly.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.