I read several posts online about evaluation metrics for classification models. Only accuracy, precision, recall, F-1 score, ROC, AUC, Confusion matrix are mentioned. However, I found a couple of Kaggle competitions use log loss as the evaluation metric. For example, Dogs vs. Cats Redux: Kernels Edition.
2
-
5$\begingroup$ It is the preferred metric and is one example of a strictly proper scoring rule. stats.stackexchange.com/questions/357466/… fharrell.com/post/class-damage fharrell.com/post/classification stats.stackexchange.com/a/359936/247274 stats.stackexchange.com/questions/464636/… twitter.com/f2harrell/status/1062424969366462473?lang=en $\endgroup$– DaveJul 10, 2021 at 5:40
-
1$\begingroup$ @Dave: do you want to post your comment(s) as an answer? Better to have a short answer than no answer at all. Anyone who has a better answer can post it. $\endgroup$– Stephan KolassaJul 10, 2021 at 8:20
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
YES, this is a reasonable evaluation metric. In particular, log loss is a strictly proper scoring rule that is maximized in expected value by the true probability values.
A standard reason to prefer a metric like accuracy is that it seems easy to interpret. "I got an accuracy of $95\%$, so that's like an $\text{A}$ in school, and I am happy." I would argue that accuracy has to be evaluated in context. A standard way that an accuracy of $95\%$ might be poor performance is if $99\%$ of the cases belong to one class, which means that you could get a higher accuracy just by predicting the majority class every time.
Consequently, I do not see accuracy as easy to interpret, and I do not buy the argument to prefer accuracy over a strictly proper scoring rule like log loss due to the ease with which accuracy can be interpreted.
-
$\begingroup$ While the log-loss can be an evaluation metric, the proper choice of evaluation metric depends on the nature of the application, so you shouldn't use the log loss as the evaluation metric if the accuracy is the correct metric for your application. I give an example of this here stats.stackexchange.com/questions/312780/… , where the use of a proper scoring rule makes a sub-optimal model choice from the perspective of the requirements of the application (because it ignores the requirements). $\endgroup$ Jun 3, 2022 at 21:22