4
$\begingroup$

Sometimes you do not have firm Y/N labels, but e.g. 80% probability of Y as a label. E.g. this happens, if you train a model on a small amount of labelled data, predict for a large amount of unlabelled data and then want to use the predictions as soft-labels.

If I want to use soft labels, then a lot of software for elastic net logistic regression (e.g. glmnet in R) does not allow labels in (0,1). Ideas for software that can handle such soft labels directly is also welcome. One potential solution that occurred to me is to use observation weights - e.g. instead of a 0.8 soft label use create two observations with weights 0.2 and 0.8 and outcome 0 and 1, respectively.

Am I assuming correctly that that would work fine?

Any other special considerations (I could only think of perhaps making sure that such pairs of observations always end top in the same fold when doing cross-validation and to certainly not split such a pair across training and test data)?

$\endgroup$
1
  • $\begingroup$ Beta regression? Through gamboostLSS potentially? $\endgroup$
    – usεr11852
    Commented Jun 19, 2019 at 17:53

1 Answer 1

1
$\begingroup$

I do not have a definite answer regarding the use of observation weights, but I don't think it's ideal. You could see if the approach works by simulating fake data and check if you can retrieve the original parameters.

If you want to continue using a GLM, you could apply a logit transform to your label and then do a standard linear regression (and transform back the output into probabilities with the inverse logit). If you don't want to transform your label, maybe this unconventional but you could keep the logit link and change the binomial distribution to a normal distribution. This should allow labels between 0 and 1, but the predictions will not be constrained (you could always apply min/max in post-processing).

Having said that, if I understood your problem correctly, I would recommend something like a mixture model. Indeed, you could write the likelihood of your model as: $$ p(x, y | \theta) = p(x, y = 1 | \theta) p(y = 1) + p(x, y = 0 | \theta)p(y=0) $$ Where $y$ is the label, $p(y=1) = 1 - p(y=0)$ the soft label for $y=1$, $x$ the features and $\theta$ the parameters. $p(x, y = 1 | \theta)$ and $p(x, y = 0 | \theta)$ are the likelihoods of the logistic regression when $y=1$ or $y=0$ respectively.

If you are feeling Bayesian, you could do this using the probabilistic programming language Stan for example. The Lasso could be implemented using a Laplace distribution for the prior $p(\theta)$. For Elastic Net, you would use a mixture of Gaussian and Laplace distribution. Alternatively, you could also use something like a horseshoe prior.

$\endgroup$
1
  • $\begingroup$ The Bayesian solution will be the most elegant and efficient. An approximate solution may be derived using multiple imputation, e.g., by creating copies of the dataset where a variable that is not known to be 0 or 1 but has an 0.8 change of being a 1 will give rise to dozens of "completed" datasets where over the datasets 0.8 of the guesses for the variable will be 1 and 0.2 will be 0. $\endgroup$ Commented Jun 26, 2019 at 11:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.