2
$\begingroup$

When does one train and cross-validate some models on a particularly objective (say, likelihood), and then evaluate or compare the models based on a different metric (say, AUC)? Any advantages and disadvantages to using different metrics, instead of using the same one which seems more natural?

Edit: Also, do we always train and (cross) validate a model on the same metric?

$\endgroup$
  • $\begingroup$ I can see choosing different metrics is desirable when a particular training metric makes it easy to train and a different evaluation metric matters more in reality or is more interpretable, but I'd like to hear others' opinions. $\endgroup$ – Lei Huang Sep 20 '18 at 18:11
2
$\begingroup$

This is one of the dirty little secrets of statistics: quite often, a model is fitted and/or selected based on one criterion, e.g., likelihood maximization, and then evaluated on a different metric, which may or may not be related to the first one.

For instance, I work in forecasting. Most papers fit models using a variant of likelihood maximization, but then rank different models (specifically: their forecasts) based on the or the .

This does not make sense.

Yes, optimizing on one KPI may be easier than optimizing on the one we will use to evaluate. For instance, likelihoods are typically nice, smooth and infinitely differentiable, and the gradients and Hessians may have nice closed forms and be easy to evaluate, whereas the MAPE or MAE are not even differentiable everywhere. However:

  1. This reminds me of the old story of the drunk that lost his keys and was searching for them under a street light - not because that was where he lost them, but because the light was better there.
  2. I almost never see a discussion of this trade-off. Very few people seem to be aware of the fact that their software uses a particular criterion to fit a model, and that this may be different from the one they later use to evaluate it.

However, one point where this does not apply is when you maximize a likelihood to fit a predictive distribution, then summarize this distribution with a single number, which is a functional of the distribution chosen to minimize a specified loss function. This is a Good Thing, and it is valid because it explicitly separates the prediction from the decision aspect.

$\endgroup$
1
$\begingroup$

Yes, it's very common.

Whenever we train a model for more than an academic exercise, we have some goal to achieve, and the model is intended asa piece of the solution to some problem. For example, let's say we are training a logistic regression on some health outcomes data, our goal is to identify those patients that for sure do not have some disease, so we can send the rest to a lab to run expensive tests that will give us a final diagnosis.

The modeling part of the exercise is concerned with determining the probability that the patient has some disease, conditional on everything that we have measured about them. Our focus in the regression stage is accurately estimating this probability. The likelihood used in training the logistic regression is designed to accomplish this task.

After we have good probabilities in hand, we have a subsequent problem of how confident we need to be to order the expensive test. This is a decision problem, not a statistical one. At this point, things like the expense of the test and the severity of the disease come into play. We want to send more records for further testing if the disease is cancer than if the disease is a common cold. For this part of the solution, things like precision and recall of the decision rule come into play.

So, just like in engineering, it's best practice to break down any problem into subproblems, and the subproblems may call for different evaluation strategies.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.