When evaluating a machine learning (or other statistical model) against multiple evaluation metrics, is there a standardized way to choose the "best" model? As a concrete example, for a two class instance segmentation model, both $F_1$ and symmetric best dice are important indicators of the model's performance, but the metrics may give different answers to which model is best. Is there an appropriate way to combine the two metrics, such as harmonic mean to combine precision and recall?

  • 1
    $\begingroup$ The short answer is no. You can finagle them together however you want, but at the end of the day if you're using a non-proper-scoring-rule evaluation, you need to design your measure for whatever decision problem you're really trying to address. $\endgroup$ May 14 at 22:54
  • 1
    $\begingroup$ I’m not even convinced that scoring rule properness is the issue. Log loss and Brier score need not give the same model as that one with the best performance. $\endgroup$
    – Dave
    May 15 at 0:04
  • $\begingroup$ Indeed, it's natural that different metrics may yield different best models. $\endgroup$
    – gunes
    May 15 at 10:14


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.