# Balancing Multiple Evaluation Metrics for a Model

When evaluating a machine learning (or other statistical model) against multiple evaluation metrics, is there a standardized way to choose the "best" model? As a concrete example, for a two class instance segmentation model, both $$F_1$$ and symmetric best dice are important indicators of the model's performance, but the metrics may give different answers to which model is best. Is there an appropriate way to combine the two metrics, such as harmonic mean to combine precision and recall?

• The short answer is no. You can finagle them together however you want, but at the end of the day if you're using a non-proper-scoring-rule evaluation, you need to design your measure for whatever decision problem you're really trying to address. May 14, 2022 at 22:54
• I’m not even convinced that scoring rule properness is the issue. Log loss and Brier score need not give the same model as that one with the best performance.
– Dave
May 15, 2022 at 0:04
• Indeed, it's natural that different metrics may yield different best models. May 15, 2022 at 10:14

It depends on what you value from your predictions. In your example, if you value a high $$F_1$$ score over a high dice score, you might be inclined to go with the model with the highest $$F_1$$ score. If you value a high dice score over a high $$F_1$$ score, you might be inclined to go with the model with the highest dice score. If you value both, you might be inclined to go with a model that never achieves the highest of either but always performs well, as opposed to the top-performing model in terms of $$F_1$$ that has a terrible dice score (or the top-performing model in terms of dice score that has a terrible $$F_1$$ score).
If you want to make this quantitative, you can use an equation of multiple measures of model performance that accounts for how you value each measure of performance. Economics refers to such an equation as a utility function, and different users can have different utility functions. In many regards, the $$F_1$$ score can be seen as a utility function of the precision and recall, so you are already used to working with this notion of combining measures of performance in a way that gives one real number as an output.