# Why is cross entropy not a common evaluation metric for model performance?

When we train a classifier, we use cross entropy as a loss function and, for example, an F-Score as an evaluation metric, but why?

Why not use cross entropy on the test set to evaluate the model performance?

Especially in a scenario where we care about the confidence of the model, it would give us a nice metric. Yet, I can't remember seeing a single paper using this. So I must be missing something.

• At least for AIC, it seems what you are missing is that the AIC is actually an asymptotic approximation to the cross entropy between the parametric distribution induced by your model and the true distribution of the data (multiplied by -2 for historic reasons). Oct 9, 2018 at 3:32
• @baruum Thanks for the comment, I removed that remark since it might lead to confusion. And you are right, for a model with O(1) parameters AIC and cross entropy are the same, modulo constants. Oct 9, 2018 at 3:51

In general, from an information theoretic point of view, binary classification with balanced classes is a "harder" problem than binary classification with a 90/10 class imbalance, as you have less information to start with (more mathematically compare $$0.1\ln 0.1 + 0.9 \ln 0.9$$ to $$2\cdot 0.5\ln 0.5$$). If you're trying to gauge to what extent your classifier is performing well for two different problems, with different class balances, you have the competing effects that perhaps one problem's features contain more information about the target variable, but the other problem is just easier to solve.
• How about $e^{CE}$? I think that'd be the geometric-mean of the absolute differences between $1.0$ and the predicted prob for the correct class? Mar 25, 2020 at 11:08