When building a learning algorithm we are looking to maximize a given evaluation metric (say accuracy), but the algorithm will try to optimize a different loss function during learning (say MSE/entropy).
Why are the evaluation metrics not used as loss functions for the learning algorithm then? Won't we then be optimizing the same metric that we are interested in?
Is there something I am missing?
It's a good question. Generally, I would argue that you should try to optimise a loss function which corresponds to the evaluation metric you care most about.
You might however want to know about other evaluation metrics.
For example, when doing classification, I'm of the opinion that you would need to give me a pretty good reason to not be optimising the cross-entropy. That said, the cross-entropy is not a very intuitive metric, so you might, once you've finished training, also want to know how good your classification accuracy is, to get a feel for whether your model is actually going to be of any real world use (it might be the best possible model and have a better cross-entropy than everybody else's, but still have insufficient accuracy to be of use in the real world).
Another argument I'm less familiar with, is, mainly in tree-based (or other greedy) algorithms, whether using certain losses mean you make better splits early on and allow you to better optimise the metric you care about globally. For example, people tend to use Gini or Information Entropy (note, not cross-entropy) when deciding on what the best split in a decision tree is. The only arguments I've ever heard for this, are not very convincing, and are basically arguments for not using accuracy but using cross-entropy instead (things around class imbalance maybe). I can think of two reasons you might use Gini when trying to get the best cross-entropy:
Something to do with local learning and greedy decision-making, as alluded to above (not convinced by this I must add).
Something to do with the actual computational implementation. In theory, a decision tree evaluates every possible split at every node and finds the best according to your criterion, but in reality, as I understand it, it does not do this and uses approximate algorithms, which I suspect leverage properties of your loss criterion.
In summary, the main reason you would have multiple evaluation metrics, is to understand what your model is doing. There might be reasons related to finding the best solution by approximate methods which mean you want to maximise metric A in order to get a solution which comes close to maximising metric B.
Often, MSE/cross-entropy are easier to optimize than for accuracy, because they are differentiable wrt to the model parameters, and in some cases, even convex, which makes it a lot easier.
Even in cases where the metric is differentiable, you might want a loss which has "better behaved" numerical properties -- see this post on the gradients of the dice-coefficient metric.
It is a good question. I would like to answer it from a different perspective. Assume your model's intent is to find out the price of a house given a host of features (location, num of bedrooms, area etc). So you train a model. Your peer and competitor in same company also develops another model. Now both of you go to the boss and ask her to use the models developed.
For the next 100 customers, she keeps close track of the predictions of both models. The one that predicts prices closest to the actual price is the model that is better and its creator needs to be promoted. But how would she evaluate the models? In all likelihood, she would choose a metric like either the MSE or the MAE. If you knew the metric before-hand, it makes sense to train your model using that same metric as a loss function, provided it is something that has a few mathematical properties needed for gradient descent. So in a sense, the loss function can be selected based on how the model is going to be evaluated.
But there could be situations where the evaluation is based on a metric that may make it difficult for the gradient descent to happen. Maybe it gives out many local minima's or maybe the gradient computation is just too expensive etc. For e.g. in case of MAE above, you may have to write additional code to take care of the learning rate as your model is approaching the minima else it will definitely overshoot...this is a minor code fix. But there could be other cases where your model may not yield the best results if the evaluation metric is directly selected as the loss function. Your best chance is to take the appropriate gradient-friendly loss metric that is most closely applicable to the problem you are trying to solve and take your chance with it. In all likelihood, your results will be much better than if you were to take some gradient un-friendly (evaluation) metric and get stuck at some local minima.
Nonetheless the question itself is highly relevant. Rather than blindly going for MSE for regression and CE for classification, one should definitely look at how the model is going to be evaluated ('used' may be a better term) and plan a loss function accordingly. In most cases a readymade pre-written loss function is available, but certain occasions could demand writing a customized loss function to suit the evaluation metric
As explained here in relation to the no free lunch theorem (https://peekaboo-vision.blogspot.com/2019/07/dont-cite-no-free-lunch-theorem.html, see the final part), “learning is impossible without proper assumptions”. If you have a model which simply minimizes the evaluation metric, you are not making any assumption about the data and what's meaningful about it. In other words, how to learn. In this situation, the model cannot learn anything, all it can do is to "memorize" the details (including the noise) of the training set, and will lack generalization power to perform well on unseen data. Basically, you need to build the model on some assumptions. This is of great generality, but in the case of ML and algorithms, those assumptions are expressed in an analytical way, encoded in the loss function. These assumptions define what's relevant about the data, allowing the model to learn something beyond the noise and to have generalizing power.
Let's see with an example of why using the metric as a loss function would not work. Let's consider a typical classification problem, where you have two classes and a 2D feature space. You want that algorithm to find a 1D boundary that crosses the 2D space that will allow you to classify new data points. Now, imagine you create that boundary by simply minimizing the misclassification rate in the training set because you reason that that's the way you will evaluate the model later. In that case, the training step trivial, since you can easily draw the boundary such that you classify correctly 100% of the training set points. Thus, you achieve your goal, a perfect result for the metric in the training step. However, that's obviously overfitting, since you have to draw the boundary without actually learning anything about the data. When you assess the model on the test set the performance will be poor.
Moreover, I think that in some algorithms, the approach doesn't make any sense at all. Think of a tree algorithm. It's based on splitting the data several times, in different steps. What's the best way to split the data at a certain step? The model is based on some analytical assumptions to split it in the most informative way. However, in which way could you split it to maximize the metric (e.g. classification accuracy), if you still need to perform possibly thousand of splitting steps before the model can make such classification?
The differences are:
A loss function is used to train your model. A metric is used to evaluate your model.
A loss function is used during the learning process. A metric is used after the learning process
Example: Assuming you train three different models each using different algorithms and loss function to solve the same image classification task. Choosing the best model based on loss error would not always work since they are not directly comparable. Therefore, metrics are used to evaluate your trained models.
In general, when loss error decreases your metric scores improve. Therefore, the two are linked and sharing the same objective.
There is no learning theoretical requirement for loss function and test-evaluation metric mismatch. An argument for such a practice however :
