Elastic net/LASSO with soft labels

Question

Sometimes you do not have firm Y/N labels, but e.g. 80% probability of Y as a label. E.g. this happens, if you train a model on a small amount of labelled data, predict for a large amount of unlabelled data and then want to use the predictions as soft-labels.

If I want to use soft labels, then a lot of software for elastic net logistic regression (e.g. glmnet in R) does not allow labels in (0,1). Ideas for software that can handle such soft labels directly is also welcome. One potential solution that occurred to me is to use observation weights - e.g. instead of a 0.8 soft label use create two observations with weights 0.2 and 0.8 and outcome 0 and 1, respectively.

Am I assuming correctly that that would work fine?

Any other special considerations (I could only think of perhaps making sure that such pairs of observations always end top in the same fold when doing cross-validation and to certainly not split such a pair across training and test data)?

Answer 1

I do not have a definite answer regarding the use of observation weights, but I don't think it's ideal. You could see if the approach works by simulating fake data and check if you can retrieve the original parameters.

If you want to continue using a GLM, you could apply a logit transform to your label and then do a standard linear regression (and transform back the output into probabilities with the inverse logit). If you don't want to transform your label, maybe this unconventional but you could keep the logit link and change the binomial distribution to a normal distribution. This should allow labels between 0 and 1, but the predictions will not be constrained (you could always apply min/max in post-processing).

Having said that, if I understood your problem correctly, I would recommend something like a mixture model. Indeed, you could write the likelihood of your model as: $$ p(x, y | \theta) = p(x, y = 1 | \theta) p(y = 1) + p(x, y = 0 | \theta)p(y=0) $$ Where $y$ is the label, $p(y=1) = 1 - p(y=0)$ the soft label for $y=1$, $x$ the features and $\theta$ the parameters. $p(x, y = 1 | \theta)$ and $p(x, y = 0 | \theta)$ are the likelihoods of the logistic regression when $y=1$ or $y=0$ respectively.

If you are feeling Bayesian, you could do this using the probabilistic programming language Stan for example. The Lasso could be implemented using a Laplace distribution for the prior $p(\theta)$. For Elastic Net, you would use a mixture of Gaussian and Laplace distribution. Alternatively, you could also use something like a horseshoe prior.