2
$\begingroup$

When building a learning algorithm we are looking to maximize a given evaluation metric (say accuracy), but the algorithm will try to optimize a different loss function during learning (say MSE/entropy).

Why are the evaluation metrics not used as loss functions for the learning algorithm then? Won't we then be optimizing the same metric that we are interested in?

Is there something I am missing?

$\endgroup$
2
$\begingroup$

It's a good question. Generally, I would argue that you should try to optimise a loss function which corresponds to the evaluation metric you care most about.

You might however want to know about other evaluation metrics.

For example, when doing classification, I'm of the opinion that you would need to give me a pretty good reason to not be optimising the cross-entropy. That said, the cross-entropy is not a very intuitive metric, so you might, once you've finished training, also want to know how good your classification accuracy is, to get a feel for whether your model is actually going to be of any real world use (it might be the best possible model and have a better cross-entropy than everybody else's, but still have insufficient accuracy to be of use in the real world).

Another argument I'm less familiar with, is, mainly in tree-based (or other greedy) algorithms, whether using certain losses mean you make better splits early on and allow you to better optimise the metric you care about globally. For example, people tend to use Gini or Information Entropy (note, not cross-entropy) when deciding on what the best split in a decision tree is. The only arguments I've ever heard for this, are not very convincing, and are basically arguments for not using accuracy but using cross-entropy instead (things around class imbalance maybe). I can think of two reasons you might use Gini when trying to get the best cross-entropy:

1: Something to do with local learning and greedy decision making, as alluded to above (not convinced by this I must add)

2: Something to do with the actual computational implementation. In theory, a decision tree evaluates every possible split at every node and finds the best according to your criterion, but in reality, as I understand it, it does not do this and uses approximate algorithms, which I suspect leverage properties of your loss criterion.

In summary, the main reason you would have multiple evaluation metrics, is to understand what your model is doing. There might be reasons related to finding the best solution by approximate methods which mean you want to maximise metric A in order to get a solution which comes close to maximising metric B.

$\endgroup$
  • $\begingroup$ So one reason might be the efficiency of the implementation (as stated also by @shimao 's answer) in using losses with "good" properties for the numeric algorithm. But the thing that puzzles me the most is how can we relate a specific loss with a different evaluation metric. Can it be that using the wrong combination of them makes the algorithm optimize a loss that worsens the metric? $\endgroup$ – Jesús Ros Nov 29 '18 at 7:39
  • $\begingroup$ yes, I agree. It seems like a bit of a leap of faith to say that "optimising Gini gets you a decent approximation to optimising logloss and the implementation is way speedier". I'm sure that formally mathematically showing this would be hard. The main place I've seen this used is in decision trees/random forests. $\endgroup$ – gazza89 Nov 30 '18 at 10:33
  • $\begingroup$ Ok, thank you @gazza89. If someone has a reference with details about that I'd be very interested. $\endgroup$ – Jesús Ros Nov 30 '18 at 16:40
1
$\begingroup$

Often, MSE/cross-entropy are easier to optimize than for accuracy, because they are differentiable wrt to the model parameters, and in some cases, even convex, which makes it a lot easier.

Even in cases where the metric is differentiable, you might want a loss which has "better behaved" numerical properties -- see this post on the gradients of the dice-coefficient metric.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.