# Semi supervised learning with partially unobservable labels

As I understood the concept of semi-supervised learning is to train a classifier on the minimal available subset of correctly labeled data in order to predict the labels of a greater previously unlabelled dataset, therefore creating pseudo-labels.

Lastly, a supervised model is trained on this dataset containing labels and pseudo labels.

But what if I just have two labels say $$\{0,1\}$$ and $$1$$ is always observable but $$0$$ cannot be observed with certainty.

An example: suppose I have to train a model that gives me the probability of converting a certain firm into a client. I can observe my current clients (therefore label=$$1$$) but for nonclients, they can either be prospects, therefore, $$1$$ or not clients at all therefore true $$0$$.

This seems like a a case of PU learning — learning from positive-only and unlabeled examples. The original paper on this is here. One of the main results is the following:

Suppose that $$y$$ is the true class label (0 or 1), $$s$$ is the observed class label (0 if negative or unlabeled, 1 if positive AND labeled), and $$x$$ is our data. Then $$p(y = 1 | x) = p(s = 1 | x) * c$$, where $$c$$ is a constant. In fact, you can show that $$c = p(s=1 | y=1)$$.

So, how do we apply this in the real world? Well, based on our dataset, we can make the following statements for all $$x$$:

1. $$p(s = 1 | y = 1; x) = \alpha$$ (assume this is a constant)
2. $$p(s = 0 | y = 1; x) = 1 - \alpha$$
3. $$p(s = 1 | y = 0; x) = 0$$ (all labeled examples must be positive)
4. $$p(s = 0 | y = 0; x) = 1$$ (you don't ever label a negative example)

Through some probabilistic manipulation, we can show that $$\alpha = \frac{1}{c}$$. So our problem is really about estimating $$\alpha$$. Suppose you had a magic function $$h(x) = p(y=1 | x)$$; i.e. you have the classifier that you want to learn. If you were to prove this in real life, one way to do this is to show that $$\mathbb{E}[h(x) | y=1] = \alpha$$. From this, we have the following algorithm:

$$\alpha \sim= \frac{1}{|V^+|} \sum_{x \in |V^+|} h(x)$$.

This means that you can naively train a model to discriminate between labeled positive examples and everything else, then rescale via $$\alpha$$ to obtain a decent estimator. That is; you don't have magical function $$h(x)$$, but the naive model behaves the same way on positive labeled examples as the ideal magical model.

Concretely, in a logistic regression toy example with $$x \in \mathbb{R}^2$$, if you draw the decision boundaries obtained from a naive vs. rescaled classifier, you will notice that this rescaling "shifts" the decision boundary linearly.