3
$\begingroup$

Set up

You have an input dataset X, and each row has multiple labels. Eg, 3 possible labels, [1,0,1] etc

Problem

The typical approach is to use BCEwithlogits loss or multi label soft margin loss. But what if the problem is now switched to all the labels must be correct, or don't predict anything at all? (Only predict if certain that all labels are correct)

What loss function do we pick for this? I thought of coding a custom loss function that returns 0 if all the labels match, else 1 but it seems “hacky”.

$\endgroup$
4
  • $\begingroup$ Such a loss function (that can only be 0 or 1) would have a gradient of zero almost everywhere making it nearly impossible to optimize. $\endgroup$
    – jodag
    Oct 16, 2020 at 22:06
  • $\begingroup$ What would you recommend? @jodag $\endgroup$
    – Wboy
    Oct 18, 2020 at 7:15
  • $\begingroup$ You’ve specified the right loss function of what you really want is a model that only predicts a category when it is absolutely certain that the category is correct. I think you don’t want that. Remember that common “classification” techniques like logistic regression and neural network predict probabilities of class membership that the user then can translate into categories, depending on thresholds and an assessment of damage caused by misclassifications. $\endgroup$
    – Dave
    Oct 18, 2020 at 7:31
  • $\begingroup$ Of possible interest: stats.stackexchange.com/questions/464636/… and the linked answers (and the links in those, like Frank Harrell’s blog $\endgroup$
    – Dave
    Oct 18, 2020 at 7:33

1 Answer 1

3
$\begingroup$

problem is now switched to all the labels must be correct, or don't predict anything at all?

You don't need any special loss function in here. You can simply use some regular loss, train the classifier to predict probabilities, and then, at inference time, make decision only if all the probabilities are higher than some threshold.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.