1
$\begingroup$

When evaluating a machine learning (or other statistical model) against multiple evaluation metrics, is there a standardized way to choose the "best" model? As a concrete example, for a two class instance segmentation model, both $F_1$ and symmetric best dice are important indicators of the model's performance, but the metrics may give different answers to which model is best. Is there an appropriate way to combine the two metrics, such as harmonic mean to combine precision and recall?

$\endgroup$
3
  • 1
    $\begingroup$ The short answer is no. You can finagle them together however you want, but at the end of the day if you're using a non-proper-scoring-rule evaluation, you need to design your measure for whatever decision problem you're really trying to address. $\endgroup$ May 14, 2022 at 22:54
  • 1
    $\begingroup$ I’m not even convinced that scoring rule properness is the issue. Log loss and Brier score need not give the same model as that one with the best performance. $\endgroup$
    – Dave
    May 15, 2022 at 0:04
  • $\begingroup$ Indeed, it's natural that different metrics may yield different best models. $\endgroup$
    – gunes
    May 15, 2022 at 10:14

1 Answer 1

1
$\begingroup$

When evaluating a machine learning (or other statistical model) against multiple evaluation metrics, is there a standardized way to choose the "best" model?

NO

It depends on what you value from your predictions. In your example, if you value a high $F_1$ score over a high dice score, you might be inclined to go with the model with the highest $F_1$ score. If you value a high dice score over a high $F_1$ score, you might be inclined to go with the model with the highest dice score. If you value both, you might be inclined to go with a model that never achieves the highest of either but always performs well, as opposed to the top-performing model in terms of $F_1$ that has a terrible dice score (or the top-performing model in terms of dice score that has a terrible $F_1$ score).

If you want to make this quantitative, you can use an equation of multiple measures of model performance that accounts for how you value each measure of performance. Economics refers to such an equation as a utility function, and different users can have different utility functions. In many regards, the $F_1$ score can be seen as a utility function of the precision and recall, so you are already used to working with this notion of combining measures of performance in a way that gives one real number as an output.

This is not even a matter of using proper scoring rules. Brier score and log loss can disagree about which model is best. If this happens, it becomes a matter of what you value for your problem. Our Stephan Kolassa has argued in favor of the log loss in answers/comments on here, while our Dikran Marsupial has written about why Brier score might be preferred.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.