3
$\begingroup$

I read several posts online about evaluation metrics for classification models. Only accuracy, precision, recall, F-1 score, ROC, AUC, Confusion matrix are mentioned. However, I found a couple of Kaggle competitions use log loss as the evaluation metric. For example, Dogs vs. Cats Redux: Kernels Edition.

$\endgroup$
2

1 Answer 1

3
$\begingroup$

YES, this is a reasonable evaluation metric. In particular, log loss is a strictly proper scoring rule that is maximized in expected value by the true probability values.

A standard reason to prefer a metric like accuracy is that it seems easy to interpret. "I got an accuracy of $95\%$, so that's like an $\text{A}$ in school, and I am happy." I would argue that accuracy has to be evaluated in context. A standard way that an accuracy of $95\%$ might be poor performance is if $99\%$ of the cases belong to one class, which means that you could get a higher accuracy just by predicting the majority class every time.

Consequently, I do not see accuracy as easy to interpret, and I do not buy the argument to prefer accuracy over a strictly proper scoring rule like log loss due to the ease with which accuracy can be interpreted.

$\endgroup$
1
  • $\begingroup$ While the log-loss can be an evaluation metric, the proper choice of evaluation metric depends on the nature of the application, so you shouldn't use the log loss as the evaluation metric if the accuracy is the correct metric for your application. I give an example of this here stats.stackexchange.com/questions/312780/… , where the use of a proper scoring rule makes a sub-optimal model choice from the perspective of the requirements of the application (because it ignores the requirements). $\endgroup$ Jun 3 at 21:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.