When we train a classifier, we use cross entropy as a loss function and, for example, an F-Score as an evaluation metric, but why?

Why not use cross entropy on the test set to evaluate the model performance?

Especially in a scenario where we care about the confidence of the model, it would give us a nice metric. Yet, I can't remember seeing a single paper using this. So I must be missing something.

  • 1
    $\begingroup$ At least for AIC, it seems what you are missing is that the AIC is actually an asymptotic approximation to the cross entropy between the parametric distribution induced by your model and the true distribution of the data (multiplied by -2 for historic reasons). $\endgroup$ – baruuum Oct 9 '18 at 3:32
  • $\begingroup$ @baruum Thanks for the comment, I removed that remark since it might lead to confusion. And you are right, for a model with O(1) parameters AIC and cross entropy are the same, modulo constants. $\endgroup$ – user695652 Oct 9 '18 at 3:51

I always use (test) cross-entropy under cross-validation to assess the performance of a classification model. It's far more robust than accuracy on small datasets (because accuracy isn't "smooth"), and far more meaningful than accuracy (although perhaps not than precision and recall) when classes are imbalanced.

However, the problem with cross-entropy, is that it doesn't live on any objective scale, it's a very relative metric. You can compare the performance of XGBoost Vs a Neural Network on a given data set and the one with a lower cross-entropy (or higher test log-likelihood) is the better model. Saying that "XGBoost gets a cross-entropy of X on problem A and a cross-entropy of Y on problem B" is harder to interpret.

In general, from an information theoretic point of view, binary classification with balanced classes is a "harder" problem than binary classification with a 90/10 class imbalance, as you have less information to start with (more mathematically compare $0.1\ln 0.1 + 0.9 \ln 0.9$ to $2\cdot 0.5\ln 0.5$). If you're trying to gauge to what extent your classifier is performing well for two different problems, with different class balances, you have the competing effects that perhaps one problem's features contain more information about the target variable, but the other problem is just easier to solve.

For that reason, you wouldn't get an academic paper (I hope anyway) which says, "we used a neural network to approach this problem for the first and got a cross-entropy of X". It would however be legit to say "usually people use neural networks to approach this problem and get a cross-entropy of X, but we used XGBoost and got a cross-entropy of Y", because then you're comparing two classifiers on the same problem

  • $\begingroup$ How about $e^{CE}$? I think that'd be the geometric-mean of the absolute differences between $1.0$ and the predicted prob for the correct class? $\endgroup$ – Narfanar Mar 25 '20 at 11:08

Cross entropy value might range from 0 to infinity depending on the dataset. It is not ancored between 0 and 1. I guess comparing peformance between diferent models and datasets might therefore be a problem.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.