# Logistic regression with censored labels

Given examples with features, each labelled 0 or 1, I can train a logistic regression model to predict the probability of the label being 1 given the features. What can I do if I have just negative and unlabelled examples? I'd be happy just to rank examples (ie determine which is more likely to be positive). This seems related to censoring, but I only found discussion of continuous variables.

More formally, denote the overall population of examples as $X=X_0\sqcup X_1$, where $X_0$ and $X_1$ are negative and positive examples respectively. Suppose I can draw samples from $X_0$, and I can draw (unlabelled) samples from $X$. Let $F$ denote examples with some particular feature values. I'm interested in $$p(X_1|F)=1-\frac{p(F|X_0)}{p(F)}p(X_0).$$ Actually I don't think I have enough information to determine $p(X_0)$, but since I only want to rank examples it's sufficient to estimate $$\frac{p(F|X_0)}{p(F)},$$ and both these terms could be estimated from my samples. In fact if I drew $n$ samples from $X_0$ and $m$ samples from $X$, pretended the latter were all positive, and trained a logistic regression model, its prediction on $F$ should be $$\frac{p(F)m}{p(F)m+p(F|X_0)n}=\frac{m}{m+np(F|X_0)/p(F)},$$ so it produces the desired ranking. However I'm not sure how robust this is (and it's not clear how the ratio $n/m$ should be chosen).

What you most probably want is Expectation Maximization (EM) algorithm. This fig nicely shows how it works: https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm#/media/File:Em_old_faithful.gif

EM works with unlabeled as well as labeled examples. In fact, it can even work with only unlabeled examples and sometimes produce reasonably good models. The downside is that, of course, it is not guaranteed to reach any correct solution, and moreover, it is non-deterministic at its heart. If you want to learn about the idea of such algorithms in general, look for Unsupervised methods or Partially Supervised methods in regression.

Your idea sounds also reasonably good, but if you think about it, most probably you will classify incorrectly a lot of unlabeled samples, while EM algorithm will probably classify them better.

In closing, surely you can derive an adaptation of logistic regression for your problem, if you are interested in theory. In practice you would most probably need to switch to another model.

• To clarify, would X and Z (in notation from that link) be respectively the negative and unlabeled examples? Wouldn't the result be a model that predicts negative labels with probability 1? If $\theta_0$ denotes the parameters describing this "always negative" model, then $p(X,0\mid\theta_0)=1$ and $p(X,Z\mid\theta_0)=0$ for $Z\neq0$, so $p(X\mid\theta_0)=1$ is maximal. – stewbasic Apr 10 '17 at 23:13
• Yes, X will be negative and Z unlabeled samples. – Linguiloce Apr 11 '17 at 10:11
• In logistic regression you assume the data to be normally distributed. The same usually goes for EM. The value $p(X,0 | \theta_0)$ thus will not be zero in any case, rather it will be a product of likelihoods for each sample $\prod_i p(x_i | \mathcal{N}(\mu_0,sigma_0))$. In this case $\theta_0$ consists of parameters $\mu_0, sigma_0$ denoting the mean and std of negative samples, and $\mu_1, sigma_1$ for positive ones. – Linguiloce Apr 11 '17 at 10:17
• Ah, I think the likelihood you are maximising is $\prod_i p($features$|$label$)$, whereas the method I've seen is to maximise $\prod_i p($label$|$features$)$. I'll need to think about whether those are equivalent... in any case accepting your answer since there is probably enough information here for me. – stewbasic Apr 11 '17 at 22:54
• Thank you @stewbasic ! Your comment reveals that you probably have one more very important for statistics question - about likelihood versus probability, or equivalently about prior and posterior probabilities. In short, those are not equivalent, but we mostly can only compute the likelihood, i.e. $p$(features|label). You can use Bayes formula to find out that $p$(label|features) = $p$(features|label)*$p$(label)/$p$(features). – Linguiloce Apr 13 '17 at 19:29

A better way to handle this situation is to use logistic regression with soft labels rather than hard labels.

Labelling an instance $x$ with the label 1 indicates your judgement that $p(X_1|F)=1$. In your situation, that's not the case.

You have a way to sample from $X_0$; for those instances, we have $p(X_0|F)=1$ and $p(X_1|F)=0$. You also have a way to sample from $X$; for those instances, we have $p(X_0|F)=p$ and $p(X_1|F)=1-p$, where is the probability that a randomly chosen element will be in $X_0$.

So, assuming that you know the class balance (the prevalence of $X_0$ vs $X_1$), this gives you a way to assign labels to all of your samples in the training set. You assign the distribution $(1,0)$ to instances sampled from $X_0$, and the distribution $(p,1-p)$ to instances sampled from $X_1$. Then, you fit a logistic regression model to this training set.

The logistic loss (cross-entropy loss; see also here) handles these soft labels fine and is not restricted to hard labels.

You might also be interested in one-class classifiers, for a different approach and one that is not limited to linear models.