With $z$ is the logit, $p \in \{1, 0\}$ is the class. Usually binary classification problem use sigmoid and cross-entropy to compute loss: $$\mathcal{L_1} = - \sum{p \log \sigma(z) + (1-p) \log (1-\sigma(z))}$$

Now suppose we scaled $y = 2p - 1 \in \{1, -1\}$.

Can we just directly push logit up when class is 1 and down when class is -1 with this loss? $$\mathcal{L_2} = - \sum{y z}$$

I have seen some code use softplus like this: $$\mathcal{L_3} = \sum{\log(1 + e^{-y z})}$$

Does it make sense to use these "simpler" losses? Is there any mathematical reasons against using them?

Could you point me to any source (paper, book, blog...) using these "simpler" losses?

  • $\begingroup$ @Sycorax The loss is for a simple binary classifier. I guess I don't really need to interpret the predicted value as probability. In test mode, the model will rank a set of items by their possibility to belong to a class. $\endgroup$
    – THN
    Sep 21, 2018 at 15:06
  • $\begingroup$ Looking again at the maths, I think I understand it all now. But this comment section is too small to write it down. I will comeback to write an answer after doing some proofs in my paper. $\endgroup$
    – THN
    Sep 24, 2018 at 5:30

1 Answer 1


The loss function measures some "goodness" or "fitness" of the model to the data. Moreover, in the context of optimization by gradient descent, the loss function must provide gradient information for the optimizer. The gradients must also be stable and informative, that is, they are not exploding, vanishing, or saturating.

From a statistical learning point of view, loss function could be built by the principle of maximum likelihood estimation (MLE), resulting in negative log-likelihood loss (NLL), which is equivalent to cross-entropy loss. The specific form of the loss function depends on the modeling of the predicted probability. For classification, the probability is modeled by sigmoid function, resulting in the form $\mathcal{L_1} $.

Directly pushing logits up and down with an unbounded loss function like $\mathcal{L_2} $ is bad, because there is no stopping threshold in optimizing, a right prediction still affects the loss the same way as a wrong prediction. On contrary, in the softplus loss $\mathcal{L_3} $, already right predictions will contribute less to the loss compared to wrong predictions.

$\mathcal{L_3} $ is actually equivalent to $\mathcal{L_1} $. The first hint is their gradients regarding the score are the same, thus their optimization dynamics are the same. In fact, one can expand their formulas and derive one from another.

  • 1
    $\begingroup$ Could you explain a little more about why L3 is equivalient to L1 $\endgroup$
    – K_Augus
    Apr 23 at 19:08
  • $\begingroup$ @K_Augus They are equivalent when L1 is the cross-entropy loss for binary classification. It can be shown by splitting the positive and negative data. You can read the appendix of this paper (arxiv.org/abs/1903.11406) for a reference. $\endgroup$
    – THN
    Apr 26 at 2:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.