16
$\begingroup$

Most algorithms use their own loss function for optimization. But these loss functions are always different from metrics used for actual evaluation. For example, for building binary classification models, log loss is normally used as the loss function, but accuracy or F1 score is used for evaluation. loss functions and evaluation metrics are not always highly correlated. So I wonder why we don't just use evaluation metrics * -1 as the loss function.

$\endgroup$
3
  • 2
    $\begingroup$ "Log loss is normally used as the loss function" sounds vague and circular. Would you mind elaborating a little? $\endgroup$
    – whuber
    Jun 2 at 21:24
  • 3
    $\begingroup$ Here, I argue in favor of using the loss function as the evaluation metric, so you're onto something in wondering why the loss function and evaluation metric would not coincide. $\endgroup$
    – Dave
    Jun 2 at 21:34
  • 5
    $\begingroup$ "Most algorithms use their own loss function for optimization." It would be pretty strange if an algorithm didn't use its own loss function for optimization. // More to the point, one might wonder whether accuracy is an informative statistic to measure at all. stats.stackexchange.com/questions/312780/… $\endgroup$
    – Sycorax
    Jun 2 at 21:58

2 Answers 2

18
$\begingroup$

Maximizing accuracy (percent of correctly examples) is the same as minimizing error rate (percent of incorrectly classified examples). For a single observation, the loss function for the error rate is always 1 (if the predicted class does not match the label) or 0 (if the predicted class matches the label). Accordingly, the derivative of this function is always 0 except at a negligible set of points where the derivative is infinite. This excludes any gradient-based optimizer from training a model, because the model parameters almost always have an update step size of 0, except for the countable number of times when the step size is infinite.

Giving up gradient information is not a good trade, because gradient descent, Newton-Raphson and similar are very effective at finding solutions which also have high accuracies, even though accuracy was not optimized directly. Examples include and regression.

Not all models are trained with gradient information. One prominent example is tree-induction methods such as random forest (however, not all trees are free of gradients; gradient-boosted trees use gradient information). These tree-based models search for good splits by optimizing some criterion, usually gini impurity, or information gain. While these models aren't optimized using gradient information, they also aren't optimized using accuracy. I suppose hypothetically you could use accuracy as a the split criterion.

$\endgroup$
0
$\begingroup$

It's because accuracy is not a proper scoring rule. You will want to consider the cost of misclassification.

Here are some more useful links:

And here the excellent posts by Frank Harrel:

Here's a counter-example where using accuracy as loss function was actually better than using the Brier-score.

You'll find that this answer is not definitive, and it depends on your actual problem if using evaluation metrics is a valid choice. I suppose it depends on the answer to the question "Does using a surrogate loss that also is a proper scoring rule reflect the learning goal of the problem well?"

$\endgroup$
3
  • 5
    $\begingroup$ I think it might help to explain why training should use a proper scoring rule. What goes wrong if we train with something that is not a proper scoring rule? What are the benefits of training with a proper scoring rule? Granted that accuracy is not a proper scoring rule, this answer still doesn't explain why we can't use accuracy as the loss function during training. I think the answer might be stronger if it were edited to explain these points. $\endgroup$
    – D.W.
    Jun 3 at 20:47
  • 3
    $\begingroup$ While accuracy is not a proper scoring rule, optimising a proper scoring rule does not necessarily maximise accuracy, so if accuracy is your performance measure of interest, proper scoring rules are not a panacea. See my answer here, stats.stackexchange.com/questions/312780/… , where using a proper scoring rule makes a sub-optimal model choice. $\endgroup$ Jun 3 at 21:19
  • 1
    $\begingroup$ What about penalized/regularized estimation? I suppose it rules out proper scoring rules (?), yet it often works better than unpenalized methods. $\endgroup$ Jun 4 at 5:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.