When evaluating a machine learning (or other statistical model) against multiple evaluation metrics, is there a standardized way to choose the "best" model? As a concrete example, for a two class instance segmentation model, both $F_1$ and symmetric best dice are important indicators of the model's performance, but the metrics may give different answers to which model is best. Is there an appropriate way to combine the two metrics, such as harmonic mean to combine precision and recall?
$\begingroup$
$\endgroup$
3
-
1$\begingroup$ The short answer is no. You can finagle them together however you want, but at the end of the day if you're using a non-proper-scoring-rule evaluation, you need to design your measure for whatever decision problem you're really trying to address. $\endgroup$– Arya McCarthyMay 14 at 22:54
-
1$\begingroup$ I’m not even convinced that scoring rule properness is the issue. Log loss and Brier score need not give the same model as that one with the best performance. $\endgroup$– DaveMay 15 at 0:04
-
$\begingroup$ Indeed, it's natural that different metrics may yield different best models. $\endgroup$– gunesMay 15 at 10:14
Add a comment
|