Why not use evaluation metrics as the loss function?

Question

Most algorithms use their own loss function for optimization. But these loss functions are always different from metrics used for actual evaluation. For example, for building binary classification models, log loss is normally used as the loss function, but accuracy or F1 score is used for evaluation. loss functions and evaluation metrics are not always highly correlated. So I wonder why we don't just use evaluation metrics * -1 as the loss function.

Answer 1

Maximizing accuracy (percent of correctly examples) is the same as minimizing error rate (percent of incorrectly classified examples). For a single observation, the loss function for the error rate is always 1 (if the predicted class does not match the label) or 0 (if the predicted class matches the label). Accordingly, the derivative of this function is always 0 except at a negligible set of points where the derivative is infinite. This excludes any gradient-based optimizer from training a model, because the model parameters almost always have an update step size of 0, except for the countable number of times when the step size is infinite.

Giving up gradient information is not a good trade, because gradient descent, Newton-Raphson and similar are very effective at finding solutions which also have high accuracies, even though accuracy was not optimized directly. Examples include neural-networks and logistic regression.

Not all models are trained with gradient information. One prominent example is tree-induction methods such as random forest (however, not all trees are free of gradients; gradient-boosted trees use gradient information). These tree-based models search for good splits by optimizing some criterion, usually gini impurity, or information gain. While these models aren't optimized using gradient information, they also aren't optimized using accuracy. I suppose hypothetically you could use accuracy as a the split criterion.

Answer 2

It's because accuracy is not a proper scoring rule. You will want to consider the cost of misclassification.

Here are some more useful links:

• Example when using accuracy as an outcome measure will lead to a wrong conclusion.
• Use of proper scoring rule when classification is required.
• The statistical part of prediction ends with outputting a distribution.

And here the excellent posts by Frank Harrel:

• Classification vs. Prediction.
• Damage Caused by Classification Accuracy and Other Discontinuous Improper Accuracy Scoring Rules

Here's a counter-example where using accuracy as loss function was actually better than using the Brier-score.

You'll find that this answer is not definitive, and it depends on your actual problem if using evaluation metrics is a valid choice. I suppose it depends on the answer to the question "Does using a surrogate loss that also is a proper scoring rule reflect the learning goal of the problem well?"