30
$\begingroup$

We know that some objective functions are easier to optimize and some are hard. And there are many loss functions that we want to use but hard to use, for example 0-1 loss. So we find some proxy loss functions to do the work. For example, we use hinge loss or logistic loss to "approximate" 0-1 loss.

Following plot is coming from Chris Bishop's PRML book. The Hinge Loss is plotted in blue, the Log Loss in red, the Square Loss in green and the 0/1 error in black.

enter image description here

I understand the reason we have such design (for hinge and logistic loss) is we want the objective function to be convex.

By looking at hinge loss and logistic loss, it penalize more on strongly misclassified instances, and interestingly, it also penalize correctly classified instances if they are weakly classified. It is a really strange design.

My question is what are the prices we need to pay by using different "proxy loss functions", such as hinge loss and logistic loss?

Improve this question
$\endgroup$
5
  • $\begingroup$ In regression, choosing squared loss is easier to do the optimization comparing to absolute value loss. But squared loss is more sensitive to outliers. So, it should be also sensitive to certain type of the data? $\endgroup$
    – Haitao Du
    Jul 7, 2016 at 13:07
  • 5
    $\begingroup$ An easier solution is to develop optimal predicted probabilities which do not require a utility function. The utility/loss function can be applied later by the actual decision maker. Classification is equivalent to making the decision for the decision maker, and requires too much knowledge up front. $\endgroup$
    – Frank Harrell
    Jul 7, 2016 at 13:09
  • 1
    $\begingroup$ @FrankHarrell Thanks, and I am using the approach you mentioned at work, where we separate prediction and business operation. However, this is still not optimized as a whole but a greedy stage-wised local solution, right? Is it a "ostrich policy"? $\endgroup$
    – Haitao Du
    Jul 7, 2016 at 13:18
  • 2
    $\begingroup$ It may not lead to optimal decisions. The loss/utility/cost function doesn't come from the model predictors. $\endgroup$
    – Frank Harrell
    Jul 7, 2016 at 14:37
  • 2
    $\begingroup$ +1. Minimizing logistic loss corresponds to maximizing binomial likelihood. Minimizing squared-error loss corresponds to maximizing Gaussian likelihood (it's just OLS regression; for 2-class classification it's actually equivalent to LDA). Do you know if minimizing hinge loss corresponds to maximizing some other likelihood? I.e. is there any probabilistic model corresponding to the hinge loss? $\endgroup$
    – amoeba
    Mar 28, 2018 at 15:49

3 Answers 3

Reset to default
19
$\begingroup$

Some of my thoughts, may not be correct though.

I understand the reason we have such design (for hinge and logistic loss) is we want the objective function to be convex.

Convexity is surely a nice property, but I think the most important reason is we want the objective function to have non-zero derivatives, so that we can make use of the derivatives to solve it. The objective function can be non-convex, in which case we often just stop at some local optima or saddle points.

and interestingly, it also penalize correctly classified instances if they are weakly classified. It is a really strange design.

I think such design sort of advises the model to not only make the right predictions, but also be confident about the predictions. If we don't want correctly classified instances to get punished, we can for example, move the hinge loss (blue) to the left by 1, so that they no longer get any loss. But I believe this often leads to worse result in practice.

what are the prices we need to pay by using different "proxy loss functions", such as hinge loss and logistic loss?

IMO by choosing different loss functions we are bringing different assumptions to the model. For example, the logistic regression loss (red) assumes a Bernoulli distribution, the MSE loss (green) assumes a Gaussian noise.

Following the least squares vs. logistic regression example in PRML, I added the hinge loss for comparison. enter image description here

As shown in the figure, hinge loss and logistic regression / cross entropy / log-likelihood / softplus have very close results, because their objective functions are close (figure below), while MSE is generally more sensitive to outliers. Hinge loss does not always have a unique solution because it's not strictly convex.

enter image description here

However one important property of hinge loss is, data points far away from the decision boundary contribute nothing to the loss, the solution will be the same with those points removed.

The remaining points are called support vectors in the context of SVM. Whereas SVM uses a regularizer term to ensure the maximum margin property and a unique solution.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thanks for the answer. Is that possible to create some demos to show the impact for different loss intuitively? Just like we show the impact by outliers of regression using squared loss vs. least absolute loss. $\endgroup$
    – Haitao Du
    Jul 27, 2016 at 13:29
  • $\begingroup$ @hxd1011 you're welcome, I'll try to add some demos later. $\endgroup$
    – dontloo
    Jul 27, 2016 at 13:41
  • 2
    $\begingroup$ Hinge loss is convex... $\endgroup$
    – Mustafa Eisa
    Jul 28, 2016 at 12:43
  • 1
    $\begingroup$ @MustafaM.Eisa right, thanks, I meant not strictly convex.. $\endgroup$
    – dontloo
    Jul 28, 2016 at 14:15
  • $\begingroup$ @dontloo great simulation! Thank you. I will also try to upload some of my simulations later. $\endgroup$
    – Haitao Du
    Jul 28, 2016 at 14:30
7
$\begingroup$

Posting a late reply, since there is a very simple answer which has not been mentioned yet.

what are the prices we need to pay by using different "proxy loss functions", such as hinge loss and logistic loss?

When you replace the non-convex 0-1 loss function by a convex surrogate (e.g hinge-loss), you are actually now solving a different problem than the one you intended to solve (which is to minimize the number of classification mistakes). So you gain computational tractability (the problem becomes convex, meaning you can solve it efficiently using tools of convex optimization), but in the general case there is actually no way to relate the error of the classifier that minimizes a "proxy" loss and the error of the classifier that minimizes the 0-1 loss. If what you truly cared about was minimizing the number of misclassifications, I argue that this really is a big price to pay.

I should mention that this statement is worst-case, in the sense that it holds for any distribution $\mathcal D$. For some "nice" distributions, there are exceptions to this rule. The key example is of data distributions that have large margins w.r.t the decision boundary - see Theorem 15.4 in Shalev-Shwartz, Shai, and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.

Improve this answer
$\endgroup$
3
$\begingroup$

Ideally your loss function should reflect actual loss incurred by business. For instance, if you're classifying damaged goods, then the loss of misclassification could be like this:

  • marking damaged goods that were not: lost profit on potential sale
  • not marking damaged goods that were damaged: cost of return processing
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.